[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:01:40 dist-test-slave-lq8l krb5kdc[28183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:01:40 dist-test-slave-lq8l krb5kdc[28183](info): set up 2 sockets
Oct 25 08:01:40 dist-test-slave-lq8l krb5kdc[28183](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20251025 08:01:42.880242 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.980s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:01:42 dist-test-slave-lq8l krb5kdc[28183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:01:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:01:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:43.015583 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:38985
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:38985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:43.122171 28199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:43.122445 28199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:43.122550 28199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:43.125854 28199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:01:43.125929 28199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:43.125968 28199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:43.126000 28199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:01:43.126039 28199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:01:43.130362 28199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:38985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:38985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:43.131501 28199 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:43.132366 28199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251025 08:01:43.138518 28199 server_base.cc:1047] running on GCE node
W20251025 08:01:43.138444 28205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:43.138444 28207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:43.138444 28204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:43.139233 28199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:43.140326 28199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:43.141534 28199 hybrid_clock.cc:648] HybridClock initialized: now 1761379303141509 us; error 48 us; skew 500 ppm
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379303, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:43.144727 28199 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:01:43.146116 28199 webserver.cc:492] Webserver started at http://127.27.132.62:41995/ using document root <none> and password file <none>
I20251025 08:01:43.146713 28199 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:43.146782 28199 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:43.147009 28199 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:43.148866 28199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "9bab2379c19047668560c201be692083"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "104b7dba8af0fee9f0787bd287ebe382"
server_key_iv: "40fcf49f6f578795f72cfcbacf8fdf7f"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.149318 28199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "9bab2379c19047668560c201be692083"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "104b7dba8af0fee9f0787bd287ebe382"
server_key_iv: "40fcf49f6f578795f72cfcbacf8fdf7f"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.152877 28199 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:01:43.155197 28214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:43.156263 28199 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:01:43.156371 28199 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "9bab2379c19047668560c201be692083"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "104b7dba8af0fee9f0787bd287ebe382"
server_key_iv: "40fcf49f6f578795f72cfcbacf8fdf7f"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.156481 28199 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:43.188715 28199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:43.191588 28199 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:43.191749 28199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:43.198822 28199 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:38985
I20251025 08:01:43.198822 28266 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:38985 every 8 connection(s)
I20251025 08:01:43.199865 28199 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:01:43.201555 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 28199
I20251025 08:01:43.201685 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:01:43.202615 28267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:43.202642 28176 external_mini_cluster.cc:1468] Setting key 3a615790a0dad4c3da5251f8adc1c9a8
I20251025 08:01:43.208663 28267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083: Bootstrap starting.
I20251025 08:01:43.210978 28267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:43.211831 28267 log.cc:826] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:43.213660 28267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083: No bootstrap required, opened a new log
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:43.216213 28267 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bab2379c19047668560c201be692083" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 38985 } }
I20251025 08:01:43.216447 28267 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:43.216543 28267 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bab2379c19047668560c201be692083, State: Initialized, Role: FOLLOWER
I20251025 08:01:43.217025 28267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bab2379c19047668560c201be692083" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 38985 } }
I20251025 08:01:43.217201 28267 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:01:43.217268 28267 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:01:43.217363 28267 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:43.218370 28267 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bab2379c19047668560c201be692083" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 38985 } }
I20251025 08:01:43.218743 28267 leader_election.cc:304] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bab2379c19047668560c201be692083; no voters:
I20251025 08:01:43.219035 28267 leader_election.cc:290] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:01:43.219132 28272 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:43.219379 28272 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [term 1 LEADER]: Becoming Leader. State: Replica: 9bab2379c19047668560c201be692083, State: Running, Role: LEADER
I20251025 08:01:43.219741 28272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bab2379c19047668560c201be692083" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 38985 } }
I20251025 08:01:43.220312 28267 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:01:43.220978 28272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bab2379c19047668560c201be692083. Latest consensus state: current_term: 1 leader_uuid: "9bab2379c19047668560c201be692083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bab2379c19047668560c201be692083" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 38985 } } }
I20251025 08:01:43.221199 28272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [sys.catalog]: This master's current role is: LEADER
I20251025 08:01:43.221345 28273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bab2379c19047668560c201be692083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bab2379c19047668560c201be692083" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 38985 } } }
I20251025 08:01:43.221410 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:43.205102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.27.132.62:38985)
1025 08:01:43.205649 (+ 547us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:43.205661 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:43.205750 (+ 89us) server_negotiation.cc:408] Connection header received
1025 08:01:43.208070 (+ 2320us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:43.208092 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:43.208448 (+ 356us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:43.208802 (+ 354us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:43.210361 (+ 1559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.211576 (+ 1215us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:43.212277 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.212475 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:43.215725 (+ 3250us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:43.215747 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:43.215759 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:43.215799 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:43.218002 (+ 2203us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.218485 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.218491 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.218496 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.218589 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.218934 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.218941 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.218945 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.219418 (+ 473us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:43.220192 (+ 774us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:43.220355 (+ 163us) server_negotiation.cc:300] Negotiation successful
1025 08:01:43.220628 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":121,"threads_started":1}
I20251025 08:01:43.221515 28273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083 [sys.catalog]: This master's current role is: LEADER
I20251025 08:01:43.221661 28281 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:01:43.224576 28281 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:01:43.229578 28281 catalog_manager.cc:1357] Generated new cluster ID: 57dd08f814a34a4c8917bee56c3280cb
I20251025 08:01:43.229651 28281 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:01:43.237360 28281 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:01:43.238227 28281 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:01:43.242910 28281 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9bab2379c19047668560c201be692083: Generated new TSK 0
I20251025 08:01:43.243803 28281 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:43.297984 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:38985
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:43.405272 28295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:43.405495 28295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:43.405555 28295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:43.408787 28295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:43.408852 28295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:43.408968 28295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:01:43.413090 28295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:38985
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:43.414145 28295 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:43.414992 28295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:43.422197 28303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:43.422197 28301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:43.422521 28295 server_base.cc:1047] running on GCE node
W20251025 08:01:43.422706 28300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:43.423085 28295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:43.423676 28295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:43.424834 28295 hybrid_clock.cc:648] HybridClock initialized: now 1761379303424821 us; error 28 us; skew 500 ppm
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379303, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:43.427477 28295 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:01:43.428483 28295 webserver.cc:492] Webserver started at http://127.27.132.1:41375/ using document root <none> and password file <none>
I20251025 08:01:43.429001 28295 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:43.429049 28295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:43.429203 28295 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:43.431221 28295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "372277814af24cceb8f4c47a078ddc12"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "8c77a3a27197bf78e26f004d19cb4e7d"
server_key_iv: "bd7e193f57e55711e733fd4abf405809"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.431696 28295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "372277814af24cceb8f4c47a078ddc12"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "8c77a3a27197bf78e26f004d19cb4e7d"
server_key_iv: "bd7e193f57e55711e733fd4abf405809"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.435021 28295 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 08:01:43.437342 28310 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:43.438401 28295 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:01:43.438555 28295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "372277814af24cceb8f4c47a078ddc12"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "8c77a3a27197bf78e26f004d19cb4e7d"
server_key_iv: "bd7e193f57e55711e733fd4abf405809"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.438679 28295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:43.453855 28295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:43.456784 28295 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:43.456939 28295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:43.457455 28295 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:01:43.458299 28295 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:43.458346 28295 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:43.458386 28295 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:43.458407 28295 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:43.467257 28295 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:35429
I20251025 08:01:43.467290 28423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:35429 every 8 connection(s)
I20251025 08:01:43.468211 28295 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:01:43.474066 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 28295
I20251025 08:01:43.474169 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:01:43.474380 28176 external_mini_cluster.cc:1468] Setting key a65d89885bbd9552c8452a6733e16457
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379303, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
I20251025 08:01:43.483505 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:43.469913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:36807 (local address 127.27.132.62:38985)
1025 08:01:43.470066 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:43.470069 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:43.470718 (+ 649us) server_negotiation.cc:408] Connection header received
1025 08:01:43.471574 (+ 856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:43.471578 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:43.471643 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:43.471728 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:43.473377 (+ 1649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.473943 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:43.474689 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.474889 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:43.477120 (+ 2231us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:43.477136 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:43.477138 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:43.477162 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:43.481352 (+ 4190us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.481985 (+ 633us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.481987 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.481988 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.482026 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.482337 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.482338 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.482340 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.482492 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:43.482562 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:43.483208 (+ 646us) server_negotiation.cc:300] Negotiation successful
1025 08:01:43.483314 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251025 08:01:43.484243 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:43.470141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:38985 (local address 127.27.132.1:36807)
1025 08:01:43.470568 (+ 427us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:43.470603 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:43.471367 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:43.471885 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:43.471893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:43.472278 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:43.473219 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:43.473233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.474067 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:43.474070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:43.474566 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:43.474574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.474796 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:43.475483 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:43.475516 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:43.476971 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:43.481497 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:43.481505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:43.481518 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:43.481889 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:43.482106 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:43.482109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:43.482113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:43.482249 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:43.482598 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:43.482606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:43.482918 (+ 312us) client_negotiation.cc:769] Sending connection context
1025 08:01:43.483190 (+ 272us) client_negotiation.cc:240] Negotiation successful
1025 08:01:43.483504 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
I20251025 08:01:43.485562 28424 heartbeater.cc:344] Connected to a master server at 127.27.132.62:38985
I20251025 08:01:43.485837 28424 heartbeater.cc:461] Registering TS with master...
I20251025 08:01:43.486372 28424 heartbeater.cc:507] Master 127.27.132.62:38985 requested a full tablet report, sending...
I20251025 08:01:43.487943 28231 ts_manager.cc:194] Registered new tserver with Master: 372277814af24cceb8f4c47a078ddc12 (127.27.132.1:35429)
I20251025 08:01:43.489158 28231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:36807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:43.519744 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:38985
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:43.624635 28431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:43.624943 28431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:43.625080 28431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:43.630326 28431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:43.630498 28431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:43.630677 28431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:01:43.636648 28431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:38985
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:43.638206 28431 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:43.639395 28431 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:43.646945 28437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:43.647051 28439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:43.646947 28436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:43.648280 28431 server_base.cc:1047] running on GCE node
I20251025 08:01:43.648732 28431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:43.649449 28431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:43.650656 28431 hybrid_clock.cc:648] HybridClock initialized: now 1761379303650634 us; error 40 us; skew 500 ppm
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379303, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:43.653650 28431 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:01:43.654942 28431 webserver.cc:492] Webserver started at http://127.27.132.2:41513/ using document root <none> and password file <none>
I20251025 08:01:43.655608 28431 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:43.655668 28431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:43.655898 28431 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:43.658174 28431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "27690dbc79ad450daecbe5591b2aa195"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "8031d07d80935fd83504589e01f1427a"
server_key_iv: "08103ce45d1b9cde2433f20ecb51a46a"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.658756 28431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "27690dbc79ad450daecbe5591b2aa195"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "8031d07d80935fd83504589e01f1427a"
server_key_iv: "08103ce45d1b9cde2433f20ecb51a46a"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.662719 28431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251025 08:01:43.665390 28446 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:43.666682 28431 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251025 08:01:43.666795 28431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "27690dbc79ad450daecbe5591b2aa195"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "8031d07d80935fd83504589e01f1427a"
server_key_iv: "08103ce45d1b9cde2433f20ecb51a46a"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.666903 28431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:43.677518 28431 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:43.680095 28431 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:43.680262 28431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:43.680787 28431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:01:43.681618 28431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:43.681666 28431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:43.681723 28431 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:43.681756 28431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:43.690353 28431 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:39259
I20251025 08:01:43.690390 28559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:39259 every 8 connection(s)
I20251025 08:01:43.691303 28431 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:01:43.695991 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 28431
I20251025 08:01:43.696084 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:01:43.696286 28176 external_mini_cluster.cc:1468] Setting key aa1bfa57aab975f21f2e72b42bdb6850
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379303, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:43.704327 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:43.693049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:46001 (local address 127.27.132.62:38985)
1025 08:01:43.693198 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:43.693203 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:43.693885 (+ 682us) server_negotiation.cc:408] Connection header received
1025 08:01:43.694709 (+ 824us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:43.694713 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:43.694770 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:43.694851 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:43.696623 (+ 1772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.697262 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:43.698083 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.698220 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:43.700844 (+ 2624us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:43.700866 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:43.700872 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:43.700906 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:43.702426 (+ 1520us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.702889 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.702895 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.702899 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.702959 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.703207 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.703213 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.703217 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.703453 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:43.703566 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:43.704006 (+ 440us) server_negotiation.cc:300] Negotiation successful
1025 08:01:43.704126 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 08:01:43.704842 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:43.693317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:38985 (local address 127.27.132.2:46001)
1025 08:01:43.693750 (+ 433us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:43.693785 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:43.694520 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:43.695012 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:43.695021 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:43.695462 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:43.696476 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:43.696491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.697405 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:43.697411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:43.697966 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:43.697979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.698213 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:43.698859 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:43.698897 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:43.700691 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:43.702539 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:43.702544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:43.702553 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:43.702783 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:43.703049 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:43.703051 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:43.703052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:43.703123 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:43.703556 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:43.703561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:43.703779 (+ 218us) client_negotiation.cc:769] Sending connection context
1025 08:01:43.703943 (+ 164us) client_negotiation.cc:240] Negotiation successful
1025 08:01:43.704143 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":118,"threads_started":1}
I20251025 08:01:43.705917 28560 heartbeater.cc:344] Connected to a master server at 127.27.132.62:38985
I20251025 08:01:43.706148 28560 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
I20251025 08:01:43.706655 28560 heartbeater.cc:507] Master 127.27.132.62:38985 requested a full tablet report, sending...
I20251025 08:01:43.707711 28231 ts_manager.cc:194] Registered new tserver with Master: 27690dbc79ad450daecbe5591b2aa195 (127.27.132.2:39259)
I20251025 08:01:43.708293 28231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:46001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:43.739028 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:38985
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:43.844383 28567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:43.844645 28567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:43.844748 28567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:43.848095 28567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:43.848171 28567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:43.848289 28567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:01:43.852420 28567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:38985
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:43.853482 28567 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:43.854321 28567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:43.860814 28575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:43.860823 28572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:43.860823 28573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:43.861231 28567 server_base.cc:1047] running on GCE node
I20251025 08:01:43.861575 28567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:43.862097 28567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:43.863247 28567 hybrid_clock.cc:648] HybridClock initialized: now 1761379303863225 us; error 35 us; skew 500 ppm
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379303, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:43.865867 28567 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:01:43.866878 28567 webserver.cc:492] Webserver started at http://127.27.132.3:40827/ using document root <none> and password file <none>
I20251025 08:01:43.867466 28567 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:43.867528 28567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:43.867714 28567 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:43.869390 28567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "3c2e313e47524bff80ff74e7edbc2a58"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "6ea6bbc1677cf817837e1f5b5f34e17a"
server_key_iv: "0ce9de1b9916d46a5fc2e511cb96dba6"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.869863 28567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "3c2e313e47524bff80ff74e7edbc2a58"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "6ea6bbc1677cf817837e1f5b5f34e17a"
server_key_iv: "0ce9de1b9916d46a5fc2e511cb96dba6"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.873198 28567 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:01:43.875295 28582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:43.876281 28567 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:01:43.876386 28567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "3c2e313e47524bff80ff74e7edbc2a58"
format_stamp: "Formatted at 2025-10-25 08:01:43 on dist-test-slave-lq8l"
server_key: "6ea6bbc1677cf817837e1f5b5f34e17a"
server_key_iv: "0ce9de1b9916d46a5fc2e511cb96dba6"
server_key_version: "encryptionkey@0"
I20251025 08:01:43.876479 28567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:43.899158 28567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:43.901940 28567 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:43.902133 28567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:43.902712 28567 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:01:43.903628 28567 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:43.903680 28567 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:43.903738 28567 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:43.903770 28567 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:43.912685 28567 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:46793
I20251025 08:01:43.912719 28695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:46793 every 8 connection(s)
I20251025 08:01:43.913638 28567 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:01:43.914582 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 28567
I20251025 08:01:43.914681 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:01:43.914917 28176 external_mini_cluster.cc:1468] Setting key 448c91eb4d56d23da9543571751ecb50
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379303, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:43.926292 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:43.915412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:40637 (local address 127.27.132.62:38985)
1025 08:01:43.915562 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:43.915566 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:43.916659 (+ 1093us) server_negotiation.cc:408] Connection header received
1025 08:01:43.917837 (+ 1178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:43.917840 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:43.917888 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:43.917944 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:43.919926 (+ 1982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.920417 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:43.921016 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.921132 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:43.923291 (+ 2159us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:43.923307 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:43.923310 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:43.923361 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:43.924621 (+ 1260us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.925069 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.925071 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.925072 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.925110 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.925392 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.925394 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.925395 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.925540 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:43.925608 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:43.926077 (+ 469us) server_negotiation.cc:300] Negotiation successful
1025 08:01:43.926169 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 08:01:43.926940 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:43.915709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:38985 (local address 127.27.132.3:40637)
1025 08:01:43.916506 (+ 797us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:43.916555 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:43.917599 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:43.918120 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:43.918135 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:43.918751 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:43.919797 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:43.919812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.920516 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:43.920518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:43.920929 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:43.920935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.921097 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:43.921684 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:43.921731 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:43.923151 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:43.924743 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:43.924748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:43.924757 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:43.924983 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:43.925211 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:43.925214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:43.925215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:43.925309 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:43.925645 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:43.925650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:43.925873 (+ 223us) client_negotiation.cc:769] Sending connection context
1025 08:01:43.926053 (+ 180us) client_negotiation.cc:240] Negotiation successful
1025 08:01:43.926267 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":132,"threads_started":1}
I20251025 08:01:43.928056 28696 heartbeater.cc:344] Connected to a master server at 127.27.132.62:38985
I20251025 08:01:43.928287 28696 heartbeater.cc:461] Registering TS with master...
I20251025 08:01:43.928853 28696 heartbeater.cc:507] Master 127.27.132.62:38985 requested a full tablet report, sending...
I20251025 08:01:43.929773 28231 ts_manager.cc:194] Registered new tserver with Master: 3c2e313e47524bff80ff74e7edbc2a58 (127.27.132.3:46793)
I20251025 08:01:43.930308 28231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:40637
I20251025 08:01:43.940702 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.132.1@KRBTEST.COM
I20251025 08:01:43.964797 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:43.956104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34774 (local address 127.27.132.1:35429)
1025 08:01:43.956434 (+ 330us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:43.956440 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:43.956471 (+ 31us) server_negotiation.cc:408] Connection header received
1025 08:01:43.956524 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:43.956529 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:43.956701 (+ 172us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:43.956830 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:43.957590 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.958306 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:43.958919 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.959084 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:43.961318 (+ 2234us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:43.961344 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:43.961357 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:43.961402 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:43.963166 (+ 1764us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.963613 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.963619 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.963620 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.963678 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.963901 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.963905 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.963907 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.964105 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:43.964298 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:43.964476 (+ 178us) server_negotiation.cc:300] Negotiation successful
1025 08:01:43.964641 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":122,"threads_started":1}
W20251025 08:01:43.965715 28338 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34774
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 25 08:01:43 dist-test-slave-lq8l krb5kdc[28183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.1@KRBTEST.COM
I20251025 08:01:43.989033 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:43.981777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.27.132.1:35429)
1025 08:01:43.981941 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:43.981945 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:43.981995 (+ 50us) server_negotiation.cc:408] Connection header received
1025 08:01:43.982122 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:43.982125 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:43.982175 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:43.982243 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:43.983004 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.983564 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:43.984146 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:43.984283 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:43.986424 (+ 2141us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:43.986450 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:43.986453 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:43.986479 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:43.987838 (+ 1359us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.988202 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.988205 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.988206 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.988242 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:43.988403 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:43.988405 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:43.988406 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:43.988539 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:43.988633 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:43.988806 (+ 173us) server_negotiation.cc:300] Negotiation successful
1025 08:01:43.988904 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251025 08:01:43.990038 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 28295
I20251025 08:01:43.996210 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 28431
I20251025 08:01:44.003903 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 28567
I20251025 08:01:44.009843 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 28199
2025-10-25T08:01:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3139 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:01:44 dist-test-slave-lq8l krb5kdc[28710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:01:44 dist-test-slave-lq8l krb5kdc[28710](info): set up 2 sockets
Oct 25 08:01:44 dist-test-slave-lq8l krb5kdc[28710](info): commencing operation
krb5kdc: starting...
W20251025 08:01:46.004472 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.972s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:01:46 dist-test-slave-lq8l krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:01:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:01:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:46.143589 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:34469
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:34469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:46.253723 28726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:46.254014 28726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:46.254117 28726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:46.257508 28726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:01:46.257589 28726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:46.257629 28726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:46.257661 28726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:01:46.257700 28726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:01:46.262210 28726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:34469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:34469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:46.263428 28726 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:46.264325 28726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:46.269924 28732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:46.270000 28726 server_base.cc:1047] running on GCE node
W20251025 08:01:46.269912 28731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:46.269893 28734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:46.270676 28726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:46.271610 28726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:46.272763 28726 hybrid_clock.cc:648] HybridClock initialized: now 1761379306272742 us; error 40 us; skew 500 ppm
Oct 25 08:01:46 dist-test-slave-lq8l krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:46.275497 28726 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:01:46.276535 28726 webserver.cc:492] Webserver started at http://127.27.132.62:46041/ using document root <none> and password file <none>
I20251025 08:01:46.277079 28726 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:46.277132 28726 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:46.277311 28726 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:46.278995 28726 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "3d5b97b12dda490ab222048f702bf82a"
format_stamp: "Formatted at 2025-10-25 08:01:46 on dist-test-slave-lq8l"
server_key: "be50ed7fada0d5fc8c27355fcc663c0d"
server_key_iv: "8208ec916222a00e5a5dd247099b847a"
server_key_version: "encryptionkey@0"
I20251025 08:01:46.279506 28726 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "3d5b97b12dda490ab222048f702bf82a"
format_stamp: "Formatted at 2025-10-25 08:01:46 on dist-test-slave-lq8l"
server_key: "be50ed7fada0d5fc8c27355fcc663c0d"
server_key_iv: "8208ec916222a00e5a5dd247099b847a"
server_key_version: "encryptionkey@0"
I20251025 08:01:46.282800 28726 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251025 08:01:46.285015 28741 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:46.285991 28726 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251025 08:01:46.286098 28726 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "3d5b97b12dda490ab222048f702bf82a"
format_stamp: "Formatted at 2025-10-25 08:01:46 on dist-test-slave-lq8l"
server_key: "be50ed7fada0d5fc8c27355fcc663c0d"
server_key_iv: "8208ec916222a00e5a5dd247099b847a"
server_key_version: "encryptionkey@0"
I20251025 08:01:46.286187 28726 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:46.301411 28726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:46.306871 28726 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:46.307029 28726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:46.314218 28726 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:34469
I20251025 08:01:46.314240 28793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:34469 every 8 connection(s)
I20251025 08:01:46.315241 28726 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:01:46.318090 28794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:46.319495 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 28726
I20251025 08:01:46.319610 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:01:46.319865 28176 external_mini_cluster.cc:1468] Setting key 947ac755878affd6a60d1f75e64c1627
I20251025 08:01:46.324179 28794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a: Bootstrap starting.
I20251025 08:01:46.326397 28794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:46.327196 28794 log.cc:826] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a: Log is configured to *not* fsync() on all Append() calls
Oct 25 08:01:46 dist-test-slave-lq8l krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:46.329202 28794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a: No bootstrap required, opened a new log
I20251025 08:01:46.331688 28794 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5b97b12dda490ab222048f702bf82a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34469 } }
I20251025 08:01:46.331875 28794 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:46.331912 28794 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5b97b12dda490ab222048f702bf82a, State: Initialized, Role: FOLLOWER
I20251025 08:01:46.332213 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:46.321298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.27.132.62:34469)
1025 08:01:46.321743 (+ 445us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:46.321753 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:46.321787 (+ 34us) server_negotiation.cc:408] Connection header received
1025 08:01:46.322295 (+ 508us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:46.322311 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:46.322633 (+ 322us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:46.322943 (+ 310us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:46.323909 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:46.324712 (+ 803us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:46.325333 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:46.325632 (+ 299us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:46.327980 (+ 2348us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:46.327997 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:46.328008 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:46.328041 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:46.329809 (+ 1768us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:46.330203 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:46.330208 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:46.330212 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:46.330287 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:46.330521 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:46.330524 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:46.330526 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:46.330841 (+ 315us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:46.331046 (+ 205us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:46.331307 (+ 261us) server_negotiation.cc:300] Negotiation successful
1025 08:01:46.331589 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":136,"threads_started":1}
I20251025 08:01:46.332384 28794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5b97b12dda490ab222048f702bf82a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34469 } }
I20251025 08:01:46.332540 28794 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:01:46.332604 28794 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:01:46.332695 28794 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:46.333640 28794 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5b97b12dda490ab222048f702bf82a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34469 } }
I20251025 08:01:46.333999 28794 leader_election.cc:304] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d5b97b12dda490ab222048f702bf82a; no voters:
I20251025 08:01:46.334309 28794 leader_election.cc:290] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:01:46.334393 28799 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:46.334571 28799 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [term 1 LEADER]: Becoming Leader. State: Replica: 3d5b97b12dda490ab222048f702bf82a, State: Running, Role: LEADER
I20251025 08:01:46.334860 28799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5b97b12dda490ab222048f702bf82a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34469 } }
I20251025 08:01:46.335522 28794 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:01:46.337014 28801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d5b97b12dda490ab222048f702bf82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5b97b12dda490ab222048f702bf82a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34469 } } }
I20251025 08:01:46.337014 28800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d5b97b12dda490ab222048f702bf82a. Latest consensus state: current_term: 1 leader_uuid: "3d5b97b12dda490ab222048f702bf82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5b97b12dda490ab222048f702bf82a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34469 } } }
I20251025 08:01:46.337188 28801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [sys.catalog]: This master's current role is: LEADER
I20251025 08:01:46.337226 28800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a [sys.catalog]: This master's current role is: LEADER
I20251025 08:01:46.337893 28805 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:01:46.340799 28805 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:01:46.345726 28805 catalog_manager.cc:1357] Generated new cluster ID: cbed014f00d44259b2e3d06fc0504e87
I20251025 08:01:46.345803 28805 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:01:46.378208 28805 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:01:46.379594 28805 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:01:46.390103 28805 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3d5b97b12dda490ab222048f702bf82a: Generated new TSK 0
I20251025 08:01:46.390969 28805 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:46.448122 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34469
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:46.555583 28822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:46.555814 28822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:46.555893 28822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:46.559403 28822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:46.559479 28822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:46.559582 28822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:01:46.563789 28822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34469
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:46.564832 28822 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:46.565691 28822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:46.572103 28830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:46.572091 28828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:46.572196 28822 server_base.cc:1047] running on GCE node
W20251025 08:01:46.572094 28827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:46.572665 28822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:46.573238 28822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:46.574383 28822 hybrid_clock.cc:648] HybridClock initialized: now 1761379306574377 us; error 34 us; skew 500 ppm
Oct 25 08:01:46 dist-test-slave-lq8l krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:46.576897 28822 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:01:46.577924 28822 webserver.cc:492] Webserver started at http://127.27.132.1:40159/ using document root <none> and password file <none>
I20251025 08:01:46.578455 28822 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:46.578512 28822 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:46.578689 28822 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:46.580436 28822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4"
format_stamp: "Formatted at 2025-10-25 08:01:46 on dist-test-slave-lq8l"
server_key: "e429368fdb4322407fa018998cb84a21"
server_key_iv: "0b48153f087dc4965c0ba7bd1cfc476e"
server_key_version: "encryptionkey@0"
I20251025 08:01:46.580911 28822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4"
format_stamp: "Formatted at 2025-10-25 08:01:46 on dist-test-slave-lq8l"
server_key: "e429368fdb4322407fa018998cb84a21"
server_key_iv: "0b48153f087dc4965c0ba7bd1cfc476e"
server_key_version: "encryptionkey@0"
I20251025 08:01:46.584250 28822 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:01:46.586454 28837 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:46.587495 28822 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:01:46.587600 28822 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4"
format_stamp: "Formatted at 2025-10-25 08:01:46 on dist-test-slave-lq8l"
server_key: "e429368fdb4322407fa018998cb84a21"
server_key_iv: "0b48153f087dc4965c0ba7bd1cfc476e"
server_key_version: "encryptionkey@0"
I20251025 08:01:46.587697 28822 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:46.602424 28822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:46.605239 28822 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:46.605423 28822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:46.605976 28822 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:01:46.606813 28822 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:46.606861 28822 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:46.606923 28822 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:46.606950 28822 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:46.616047 28822 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:39055
I20251025 08:01:46.616098 28950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:39055 every 8 connection(s)
I20251025 08:01:46.616984 28822 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:01:46.624010 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 28822
I20251025 08:01:46.624118 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:01:46.624341 28176 external_mini_cluster.cc:1468] Setting key ce031ca5f169086a558a32b3a692600b
Oct 25 08:01:46 dist-test-slave-lq8l krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:46.629338 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:46.618662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:35437 (local address 127.27.132.62:34469)
1025 08:01:46.618841 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:46.618846 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:46.619526 (+ 680us) server_negotiation.cc:408] Connection header received
1025 08:01:46.620329 (+ 803us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:46.620332 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:46.620381 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:46.620455 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:46.622095 (+ 1640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:46.622593 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:46.623180 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:46.623304 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:46.625610 (+ 2306us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:46.625627 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:46.625630 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:46.625652 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:46.627375 (+ 1723us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:46.627989 (+ 614us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:46.627992 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:46.627993 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:46.628034 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:46.628327 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:46.628329 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:46.628330 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:46.628468 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:46.628537 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:46.629118 (+ 581us) server_negotiation.cc:300] Negotiation successful
1025 08:01:46.629218 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20251025 08:01:46.630338 28953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:46.618930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34469 (local address 127.27.132.1:35437)
1025 08:01:46.619392 (+ 462us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:46.619428 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:46.620150 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:46.620616 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:46.620624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:46.621027 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:46.621959 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:46.621973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:46.622683 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:46.622693 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:46.623098 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:46.623103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:46.623276 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:46.623920 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:46.623944 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:46.625452 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:46.627517 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:46.627526 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:46.627541 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:46.627891 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:46.628117 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:46.628120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:46.628123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:46.628248 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:46.628570 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:46.628581 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:46.628873 (+ 292us) client_negotiation.cc:769] Sending connection context
1025 08:01:46.629110 (+ 237us) client_negotiation.cc:240] Negotiation successful
1025 08:01:46.629419 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20251025 08:01:46.631565 28951 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34469
I20251025 08:01:46.631810 28951 heartbeater.cc:461] Registering TS with master...
I20251025 08:01:46.632306 28951 heartbeater.cc:507] Master 127.27.132.62:34469 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
I20251025 08:01:46.633888 28758 ts_manager.cc:194] Registered new tserver with Master: c6d46eb04cbb44ffb7b15ef0845eeef4 (127.27.132.1:39055)
I20251025 08:01:46.635073 28758 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:35437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:46.668346 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34469
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:46.777842 28958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:46.778075 28958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:46.778182 28958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:46.781515 28958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:46.781590 28958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:46.781677 28958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:01:46.786027 28958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34469
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:46.787148 28958 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:46.787987 28958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:46.794744 28963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:46.794737 28966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:46.794737 28964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:46.794972 28958 server_base.cc:1047] running on GCE node
I20251025 08:01:46.795403 28958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:46.795974 28958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:46.797142 28958 hybrid_clock.cc:648] HybridClock initialized: now 1761379306797117 us; error 37 us; skew 500 ppm
Oct 25 08:01:46 dist-test-slave-lq8l krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:46.799853 28958 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:01:46.800930 28958 webserver.cc:492] Webserver started at http://127.27.132.2:36145/ using document root <none> and password file <none>
I20251025 08:01:46.801492 28958 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:46.801545 28958 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:46.801735 28958 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:46.803542 28958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "b444bc46d0ff4a5da4cde73343a0ca7d"
format_stamp: "Formatted at 2025-10-25 08:01:46 on dist-test-slave-lq8l"
server_key: "fc2214a960bf466fdb1ea55886770216"
server_key_iv: "1063be97cbe8264a9a48d88076ba4480"
server_key_version: "encryptionkey@0"
I20251025 08:01:46.804026 28958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "b444bc46d0ff4a5da4cde73343a0ca7d"
format_stamp: "Formatted at 2025-10-25 08:01:46 on dist-test-slave-lq8l"
server_key: "fc2214a960bf466fdb1ea55886770216"
server_key_iv: "1063be97cbe8264a9a48d88076ba4480"
server_key_version: "encryptionkey@0"
I20251025 08:01:46.807547 28958 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 08:01:46.809971 28973 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:46.811137 28958 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:01:46.811261 28958 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "b444bc46d0ff4a5da4cde73343a0ca7d"
format_stamp: "Formatted at 2025-10-25 08:01:46 on dist-test-slave-lq8l"
server_key: "fc2214a960bf466fdb1ea55886770216"
server_key_iv: "1063be97cbe8264a9a48d88076ba4480"
server_key_version: "encryptionkey@0"
I20251025 08:01:46.811502 28958 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:46.821914 28958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:46.825098 28958 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:46.825289 28958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:46.825906 28958 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:01:46.826797 28958 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:46.826848 28958 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:46.826887 28958 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:46.826920 28958 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:46.836406 28958 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:46397
I20251025 08:01:46.836475 29086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:46397 every 8 connection(s)
I20251025 08:01:46.837402 28958 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:01:46.844249 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 28958
I20251025 08:01:46.844357 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:01:46.844583 28176 external_mini_cluster.cc:1468] Setting key d6083e834a956c45f1348f72ac5d283c
Oct 25 08:01:46 dist-test-slave-lq8l krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:46.850586 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:46.839271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:52419 (local address 127.27.132.62:34469)
1025 08:01:46.839532 (+ 261us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:46.839536 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:46.840231 (+ 695us) server_negotiation.cc:408] Connection header received
1025 08:01:46.841119 (+ 888us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:46.841123 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:46.841187 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:46.841268 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:46.843000 (+ 1732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:46.843576 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:46.844226 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:46.844407 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:46.846712 (+ 2305us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:46.846728 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:46.846731 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:46.846755 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:46.848446 (+ 1691us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:46.848954 (+ 508us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:46.848961 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:46.848966 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:46.849023 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:46.849276 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:46.849282 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:46.849288 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:46.849533 (+ 245us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:46.849649 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:46.850167 (+ 518us) server_negotiation.cc:300] Negotiation successful
1025 08:01:46.850317 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20251025 08:01:46.851080 29089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:46.839631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34469 (local address 127.27.132.2:52419)
1025 08:01:46.840074 (+ 443us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:46.840112 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:46.840918 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:46.841442 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:46.841451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:46.841889 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:46.842840 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:46.842854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:46.843702 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:46.843705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:46.844110 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:46.844117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:46.844348 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:46.845025 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:46.845068 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:46.846559 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:46.848588 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:46.848593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:46.848603 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:46.848856 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:46.849116 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:46.849118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:46.849120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:46.849197 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:46.849655 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:46.849661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:46.849906 (+ 245us) client_negotiation.cc:769] Sending connection context
1025 08:01:46.850091 (+ 185us) client_negotiation.cc:240] Negotiation successful
1025 08:01:46.850320 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
I20251025 08:01:46.853044 29087 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34469
I20251025 08:01:46.853333 29087 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
I20251025 08:01:46.854033 29087 heartbeater.cc:507] Master 127.27.132.62:34469 requested a full tablet report, sending...
I20251025 08:01:46.855391 28758 ts_manager.cc:194] Registered new tserver with Master: b444bc46d0ff4a5da4cde73343a0ca7d (127.27.132.2:46397)
I20251025 08:01:46.856010 28758 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:52419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:46.892395 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34469
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:47.001144 29094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:47.001367 29094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:47.001426 29094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:47.004648 29094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:47.004719 29094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:47.004801 29094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:01:47.009117 29094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34469
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:47.010195 29094 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:47.011049 29094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:47.017493 29100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:47.017498 29102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:47.017611 29099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:47.017808 29094 server_base.cc:1047] running on GCE node
I20251025 08:01:47.018184 29094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:47.018719 29094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:47.019863 29094 hybrid_clock.cc:648] HybridClock initialized: now 1761379307019844 us; error 34 us; skew 500 ppm
Oct 25 08:01:47 dist-test-slave-lq8l krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:47.022447 29094 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:01:47.023509 29094 webserver.cc:492] Webserver started at http://127.27.132.3:45859/ using document root <none> and password file <none>
I20251025 08:01:47.024051 29094 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:47.024107 29094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:47.024284 29094 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:47.025942 29094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "67b1053f828641abab77d0c79cc666a0"
format_stamp: "Formatted at 2025-10-25 08:01:47 on dist-test-slave-lq8l"
server_key: "845aafe5674a04b27111424b5038a2c5"
server_key_iv: "d58d2fd41f609fbaf63dfb35a0d0ca94"
server_key_version: "encryptionkey@0"
I20251025 08:01:47.026404 29094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "67b1053f828641abab77d0c79cc666a0"
format_stamp: "Formatted at 2025-10-25 08:01:47 on dist-test-slave-lq8l"
server_key: "845aafe5674a04b27111424b5038a2c5"
server_key_iv: "d58d2fd41f609fbaf63dfb35a0d0ca94"
server_key_version: "encryptionkey@0"
I20251025 08:01:47.029733 29094 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20251025 08:01:47.031920 29109 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:47.032886 29094 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:01:47.032990 29094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "67b1053f828641abab77d0c79cc666a0"
format_stamp: "Formatted at 2025-10-25 08:01:47 on dist-test-slave-lq8l"
server_key: "845aafe5674a04b27111424b5038a2c5"
server_key_iv: "d58d2fd41f609fbaf63dfb35a0d0ca94"
server_key_version: "encryptionkey@0"
I20251025 08:01:47.033080 29094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:47.046257 29094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:47.049325 29094 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:47.049523 29094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:47.050092 29094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:01:47.050997 29094 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:47.051048 29094 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:47.051110 29094 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:47.051136 29094 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:47.060403 29094 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:46863
I20251025 08:01:47.060499 29222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:46863 every 8 connection(s)
I20251025 08:01:47.061568 29094 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:01:47.068346 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 29094
I20251025 08:01:47.068495 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:01:47.068761 28176 external_mini_cluster.cc:1468] Setting key ae7085cf4d602e985b3b68617a1288ef
Oct 25 08:01:47 dist-test-slave-lq8l krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:47.074029 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.063448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:40179 (local address 127.27.132.62:34469)
1025 08:01:47.063617 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:47.063622 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:47.064378 (+ 756us) server_negotiation.cc:408] Connection header received
1025 08:01:47.065196 (+ 818us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:47.065199 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:47.065248 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:47.065325 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:47.067071 (+ 1746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.067631 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.068293 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.068496 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.070943 (+ 2447us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:47.070956 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:47.070958 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:47.070982 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:47.072429 (+ 1447us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:47.072885 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:47.072888 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:47.072889 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:47.072926 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:47.073135 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:47.073137 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:47.073138 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:47.073278 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:47.073346 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:47.073800 (+ 454us) server_negotiation.cc:300] Negotiation successful
1025 08:01:47.073908 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251025 08:01:47.074714 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.063767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34469 (local address 127.27.132.3:40179)
1025 08:01:47.064218 (+ 451us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:47.064256 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:47.065009 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:47.065492 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:47.065506 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:47.065946 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:47.066893 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.066907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.067751 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.067754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:47.068191 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.068198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.068374 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.069092 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:47.069131 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:47.070796 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:47.072549 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:47.072554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:47.072564 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:47.072805 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:47.072989 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:47.072991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:47.072993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:47.073072 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:47.073348 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:47.073353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:47.073587 (+ 234us) client_negotiation.cc:769] Sending connection context
1025 08:01:47.073776 (+ 189us) client_negotiation.cc:240] Negotiation successful
1025 08:01:47.074001 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20251025 08:01:47.075778 29223 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34469
I20251025 08:01:47.076012 29223 heartbeater.cc:461] Registering TS with master...
I20251025 08:01:47.076514 29223 heartbeater.cc:507] Master 127.27.132.62:34469 requested a full tablet report, sending...
I20251025 08:01:47.077507 28758 ts_manager.cc:194] Registered new tserver with Master: 67b1053f828641abab77d0c79cc666a0 (127.27.132.3:46863)
I20251025 08:01:47.078060 28758 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:40179
I20251025 08:01:47.082598 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 08:01:47.091975 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.085516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.27.132.62:34469)
1025 08:01:47.085663 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:47.085667 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:47.085783 (+ 116us) server_negotiation.cc:408] Connection header received
1025 08:01:47.085906 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:47.085908 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:47.085953 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:47.086030 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:47.086878 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.087422 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.088073 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.088289 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.089288 (+ 999us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:47.089304 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:47.089306 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:47.089327 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:47.090495 (+ 1168us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:47.090873 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:47.090876 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:47.090878 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:47.090918 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:47.091120 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:47.091123 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:47.091125 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:47.091292 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:47.091476 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:47.091690 (+ 214us) server_negotiation.cc:300] Negotiation successful
1025 08:01:47.091812 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 08:01:47.095840 28758 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:01:47.098141 28758 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:01:47.110870 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.106048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:39055 (local address 127.0.0.1:55798)
1025 08:01:47.106668 (+ 620us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:47.106709 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:47.106853 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:47.107675 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:47.107684 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:47.107708 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:47.107993 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.108001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.109390 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.109393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:47.110295 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.110304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.110436 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.110495 (+ 59us) client_negotiation.cc:769] Sending connection context
1025 08:01:47.110590 (+ 95us) client_negotiation.cc:240] Negotiation successful
1025 08:01:47.110680 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":78,"threads_started":1}
I20251025 08:01:47.110879 29237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.106540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:46863 (local address 127.0.0.1:60096)
1025 08:01:47.107249 (+ 709us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:47.107265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:47.107386 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:47.108029 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:47.108033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:47.108071 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:47.108355 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.108363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.109615 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.109618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:47.110384 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.110395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.110499 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.110518 (+ 19us) client_negotiation.cc:769] Sending connection context
1025 08:01:47.110590 (+ 72us) client_negotiation.cc:240] Negotiation successful
1025 08:01:47.110683 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"spinlock_wait_cycles":6912,"thread_start_us":109,"threads_started":1}
I20251025 08:01:47.111726 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.106201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.27.132.1:39055)
1025 08:01:47.107133 (+ 932us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:47.107140 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:47.107163 (+ 23us) server_negotiation.cc:408] Connection header received
1025 08:01:47.107258 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:47.107266 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:47.107484 (+ 218us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:47.108115 (+ 631us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:47.108230 (+ 115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.109287 (+ 1057us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.110773 (+ 1486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.111364 (+ 591us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.111498 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:47.111551 (+ 53us) server_negotiation.cc:300] Negotiation successful
1025 08:01:47.111621 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":109,"threads_started":1}
I20251025 08:01:47.111944 29239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.106830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.27.132.3:46863)
1025 08:01:47.107160 (+ 330us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:47.107166 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:47.107492 (+ 326us) server_negotiation.cc:408] Connection header received
1025 08:01:47.107580 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:47.107585 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:47.107742 (+ 157us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:47.107882 (+ 140us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:47.108483 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.109508 (+ 1025us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.110773 (+ 1265us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.111547 (+ 774us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.111647 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:47.111718 (+ 71us) server_negotiation.cc:300] Negotiation successful
1025 08:01:47.111800 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":139,"threads_started":1}
I20251025 08:01:47.112370 29238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.106830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.27.132.2:46397)
1025 08:01:47.107486 (+ 656us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:47.107492 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:47.107513 (+ 21us) server_negotiation.cc:408] Connection header received
1025 08:01:47.107583 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:47.107587 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:47.107742 (+ 155us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:47.107881 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:47.108512 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.109202 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.111485 (+ 2283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.112000 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.112097 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:47.112159 (+ 62us) server_negotiation.cc:300] Negotiation successful
1025 08:01:47.112244 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":105,"threads_started":1}
I20251025 08:01:47.112617 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.106255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:46397 (local address 127.0.0.1:40120)
1025 08:01:47.106824 (+ 569us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:47.106837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:47.106907 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:47.108029 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:47.108033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:47.108066 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:47.108355 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.108363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.109316 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.109319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:47.110295 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.110304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.110443 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.110495 (+ 52us) client_negotiation.cc:769] Sending connection context
1025 08:01:47.110590 (+ 95us) client_negotiation.cc:240] Negotiation successful
1025 08:01:47.110680 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"spinlock_wait_cycles":9344,"thread_start_us":93,"threads_started":1}
I20251025 08:01:47.114558 29157 tablet_service.cc:1505] Processing CreateTablet for tablet a824a7c6c0de4a61b5b4aef7eeb381f0 (DEFAULT_TABLE table=test-table [id=40222dfff5494e6085267a21a6bf99f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:47.114624 28885 tablet_service.cc:1505] Processing CreateTablet for tablet a824a7c6c0de4a61b5b4aef7eeb381f0 (DEFAULT_TABLE table=test-table [id=40222dfff5494e6085267a21a6bf99f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:47.114746 29021 tablet_service.cc:1505] Processing CreateTablet for tablet a824a7c6c0de4a61b5b4aef7eeb381f0 (DEFAULT_TABLE table=test-table [id=40222dfff5494e6085267a21a6bf99f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:47.115374 29157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a824a7c6c0de4a61b5b4aef7eeb381f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:47.115506 28885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a824a7c6c0de4a61b5b4aef7eeb381f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:47.115506 29021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a824a7c6c0de4a61b5b4aef7eeb381f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:47.120841 29240 tablet_bootstrap.cc:492] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0: Bootstrap starting.
I20251025 08:01:47.122148 29241 tablet_bootstrap.cc:492] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d: Bootstrap starting.
I20251025 08:01:47.122407 29242 tablet_bootstrap.cc:492] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4: Bootstrap starting.
I20251025 08:01:47.123306 29240 tablet_bootstrap.cc:654] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:47.124071 29242 tablet_bootstrap.cc:654] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:47.124184 29241 tablet_bootstrap.cc:654] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:47.124169 29240 log.cc:826] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:47.124784 29242 log.cc:826] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:47.125078 29241 log.cc:826] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:47.126223 29240 tablet_bootstrap.cc:492] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0: No bootstrap required, opened a new log
I20251025 08:01:47.126416 29240 ts_tablet_manager.cc:1403] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20251025 08:01:47.126427 29242 tablet_bootstrap.cc:492] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4: No bootstrap required, opened a new log
I20251025 08:01:47.126595 29242 ts_tablet_manager.cc:1403] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251025 08:01:47.126861 29241 tablet_bootstrap.cc:492] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d: No bootstrap required, opened a new log
I20251025 08:01:47.127033 29241 ts_tablet_manager.cc:1403] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 08:01:47.129715 29240 raft_consensus.cc:359] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 39055 } } peers { permanent_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 46397 } } peers { permanent_uuid: "67b1053f828641abab77d0c79cc666a0" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46863 } }
I20251025 08:01:47.129912 29240 raft_consensus.cc:385] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:47.129990 29240 raft_consensus.cc:740] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b1053f828641abab77d0c79cc666a0, State: Initialized, Role: FOLLOWER
I20251025 08:01:47.130026 29242 raft_consensus.cc:359] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 39055 } } peers { permanent_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 46397 } } peers { permanent_uuid: "67b1053f828641abab77d0c79cc666a0" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46863 } }
I20251025 08:01:47.130326 29242 raft_consensus.cc:385] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:47.130456 29242 raft_consensus.cc:740] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d46eb04cbb44ffb7b15ef0845eeef4, State: Initialized, Role: FOLLOWER
I20251025 08:01:47.130465 29240 consensus_queue.cc:260] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 39055 } } peers { permanent_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 46397 } } peers { permanent_uuid: "67b1053f828641abab77d0c79cc666a0" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46863 } }
I20251025 08:01:47.130618 29241 raft_consensus.cc:359] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 39055 } } peers { permanent_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 46397 } } peers { permanent_uuid: "67b1053f828641abab77d0c79cc666a0" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46863 } }
I20251025 08:01:47.130895 29241 raft_consensus.cc:385] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:47.130961 29241 raft_consensus.cc:740] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b444bc46d0ff4a5da4cde73343a0ca7d, State: Initialized, Role: FOLLOWER
I20251025 08:01:47.130976 29242 consensus_queue.cc:260] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 39055 } } peers { permanent_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 46397 } } peers { permanent_uuid: "67b1053f828641abab77d0c79cc666a0" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46863 } }
I20251025 08:01:47.131433 29240 ts_tablet_manager.cc:1434] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20251025 08:01:47.131395 29241 consensus_queue.cc:260] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 39055 } } peers { permanent_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 46397 } } peers { permanent_uuid: "67b1053f828641abab77d0c79cc666a0" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46863 } }
I20251025 08:01:47.131717 28951 heartbeater.cc:499] Master 127.27.132.62:34469 was elected leader, sending a full tablet report...
I20251025 08:01:47.131868 29242 ts_tablet_manager.cc:1434] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251025 08:01:47.132056 29087 heartbeater.cc:499] Master 127.27.132.62:34469 was elected leader, sending a full tablet report...
I20251025 08:01:47.132184 29241 ts_tablet_manager.cc:1434] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251025 08:01:47.133040 29223 heartbeater.cc:499] Master 127.27.132.62:34469 was elected leader, sending a full tablet report...
I20251025 08:01:47.163733 29248 raft_consensus.cc:493] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:01:47.163949 29248 raft_consensus.cc:515] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 39055 } } peers { permanent_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 46397 } } peers { permanent_uuid: "67b1053f828641abab77d0c79cc666a0" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46863 } }
I20251025 08:01:47.165382 29248 leader_election.cc:290] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6d46eb04cbb44ffb7b15ef0845eeef4 (127.27.132.1:39055), 67b1053f828641abab77d0c79cc666a0 (127.27.132.3:46863)
I20251025 08:01:47.168720 29089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.165568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:46863 (local address 127.27.132.2:53455)
1025 08:01:47.165795 (+ 227us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:47.165813 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:47.165916 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:47.166295 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:47.166299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:47.166321 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:47.166623 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.166630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.167649 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.167655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:47.168306 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.168316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.168429 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.168459 (+ 30us) client_negotiation.cc:769] Sending connection context
1025 08:01:47.168514 (+ 55us) client_negotiation.cc:240] Negotiation successful
1025 08:01:47.168580 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20251025 08:01:47.168938 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.165566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:39055 (local address 127.27.132.2:57713)
1025 08:01:47.166323 (+ 757us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:47.166344 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:47.166465 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:47.166729 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:47.166733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:47.166752 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:47.166965 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.166971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.167952 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.167956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:47.168547 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:47.168556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.168668 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.168687 (+ 19us) client_negotiation.cc:769] Sending connection context
1025 08:01:47.168739 (+ 52us) client_negotiation.cc:240] Negotiation successful
1025 08:01:47.168797 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":422,"thread_start_us":120,"threads_started":1}
I20251025 08:01:47.169239 29239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.165806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:53455 (local address 127.27.132.3:46863)
1025 08:01:47.165974 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:47.165979 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:47.166006 (+ 27us) server_negotiation.cc:408] Connection header received
1025 08:01:47.166073 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:47.166079 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:47.166143 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:47.166231 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:47.166757 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.167515 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.168437 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.168952 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.168999 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:47.169061 (+ 62us) server_negotiation.cc:300] Negotiation successful
1025 08:01:47.169137 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251025 08:01:47.169426 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.165790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:57713 (local address 127.27.132.1:39055)
1025 08:01:47.165990 (+ 200us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:47.165996 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:47.166426 (+ 430us) server_negotiation.cc:408] Connection header received
1025 08:01:47.166577 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:47.166580 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:47.166623 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:47.166693 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:47.167073 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.167814 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.168675 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.169174 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.169212 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:47.169261 (+ 49us) server_negotiation.cc:300] Negotiation successful
1025 08:01:47.169306 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20251025 08:01:47.169843 29177 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a824a7c6c0de4a61b5b4aef7eeb381f0" candidate_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b1053f828641abab77d0c79cc666a0" is_pre_election: true
I20251025 08:01:47.169973 28905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a824a7c6c0de4a61b5b4aef7eeb381f0" candidate_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4" is_pre_election: true
I20251025 08:01:47.170115 29177 raft_consensus.cc:2468] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b444bc46d0ff4a5da4cde73343a0ca7d in term 0.
I20251025 08:01:47.170214 28905 raft_consensus.cc:2468] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b444bc46d0ff4a5da4cde73343a0ca7d in term 0.
I20251025 08:01:47.170593 28974 leader_election.cc:304] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b1053f828641abab77d0c79cc666a0, b444bc46d0ff4a5da4cde73343a0ca7d; no voters:
I20251025 08:01:47.170854 29248 raft_consensus.cc:2804] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:01:47.170921 29248 raft_consensus.cc:493] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:01:47.170953 29248 raft_consensus.cc:3060] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:47.171947 29248 raft_consensus.cc:515] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 39055 } } peers { permanent_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 46397 } } peers { permanent_uuid: "67b1053f828641abab77d0c79cc666a0" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46863 } }
I20251025 08:01:47.172318 29248 leader_election.cc:290] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [CANDIDATE]: Term 1 election: Requested vote from peers c6d46eb04cbb44ffb7b15ef0845eeef4 (127.27.132.1:39055), 67b1053f828641abab77d0c79cc666a0 (127.27.132.3:46863)
I20251025 08:01:47.172668 28905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a824a7c6c0de4a61b5b4aef7eeb381f0" candidate_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4"
I20251025 08:01:47.172691 29177 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a824a7c6c0de4a61b5b4aef7eeb381f0" candidate_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b1053f828641abab77d0c79cc666a0"
I20251025 08:01:47.172776 28905 raft_consensus.cc:3060] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:47.172809 29177 raft_consensus.cc:3060] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:47.173734 29177 raft_consensus.cc:2468] T a824a7c6c0de4a61b5b4aef7eeb381f0 P 67b1053f828641abab77d0c79cc666a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b444bc46d0ff4a5da4cde73343a0ca7d in term 1.
I20251025 08:01:47.173738 28905 raft_consensus.cc:2468] T a824a7c6c0de4a61b5b4aef7eeb381f0 P c6d46eb04cbb44ffb7b15ef0845eeef4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b444bc46d0ff4a5da4cde73343a0ca7d in term 1.
I20251025 08:01:47.174049 28976 leader_election.cc:304] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b444bc46d0ff4a5da4cde73343a0ca7d, c6d46eb04cbb44ffb7b15ef0845eeef4; no voters:
I20251025 08:01:47.174224 29248 raft_consensus.cc:2804] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:47.174427 29248 raft_consensus.cc:697] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [term 1 LEADER]: Becoming Leader. State: Replica: b444bc46d0ff4a5da4cde73343a0ca7d, State: Running, Role: LEADER
I20251025 08:01:47.174724 29248 consensus_queue.cc:237] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 39055 } } peers { permanent_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 46397 } } peers { permanent_uuid: "67b1053f828641abab77d0c79cc666a0" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46863 } }
I20251025 08:01:47.177846 28757 catalog_manager.cc:5649] T a824a7c6c0de4a61b5b4aef7eeb381f0 P b444bc46d0ff4a5da4cde73343a0ca7d reported cstate change: term changed from 0 to 1, leader changed from <none> to b444bc46d0ff4a5da4cde73343a0ca7d (127.27.132.2). New cstate: current_term: 1 leader_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d46eb04cbb44ffb7b15ef0845eeef4" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 39055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b444bc46d0ff4a5da4cde73343a0ca7d" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 46397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67b1053f828641abab77d0c79cc666a0" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46863 } health_report { overall_health: UNKNOWN } } }
Oct 25 08:01:47 dist-test-slave-lq8l krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.1@KRBTEST.COM
I20251025 08:01:47.192914 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.184780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55802 (local address 127.27.132.1:39055)
1025 08:01:47.184959 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:47.184963 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:47.185024 (+ 61us) server_negotiation.cc:408] Connection header received
1025 08:01:47.185129 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:47.185131 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:47.185175 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:47.185245 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:47.185999 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.186485 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.187047 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.187182 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.189448 (+ 2266us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:47.189486 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:47.189499 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:47.189529 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:47.191365 (+ 1836us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:47.191738 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:47.191742 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:47.191744 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:47.191799 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:47.192030 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:47.192036 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:47.192038 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:47.192242 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:47.192459 (+ 217us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:47.192621 (+ 162us) server_negotiation.cc:300] Negotiation successful
1025 08:01:47.192749 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Oct 25 08:01:47 dist-test-slave-lq8l krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 25 08:01:47 dist-test-slave-lq8l krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.132.1@KRBTEST.COM
I20251025 08:01:47.217025 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.209425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55816 (local address 127.27.132.1:39055)
1025 08:01:47.209586 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:47.209590 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:47.209689 (+ 99us) server_negotiation.cc:408] Connection header received
1025 08:01:47.209803 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:47.209806 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:47.209853 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:47.209927 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:47.210710 (+ 783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.211182 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.211926 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.212059 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.214195 (+ 2136us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:47.214214 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:47.214217 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:47.214245 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:47.215623 (+ 1378us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:47.215987 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:47.215990 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:47.215991 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:47.216032 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:47.216214 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:47.216220 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:47.216224 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:47.216391 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:47.216536 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:47.216738 (+ 202us) server_negotiation.cc:300] Negotiation successful
1025 08:01:47.216834 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Oct 25 08:01:47 dist-test-slave-lq8l krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 25 08:01:47 dist-test-slave-lq8l krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.1@KRBTEST.COM
I20251025 08:01:47.241109 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:47.232972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.27.132.1:39055)
1025 08:01:47.233150 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:47.233156 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:47.233245 (+ 89us) server_negotiation.cc:408] Connection header received
1025 08:01:47.233409 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:47.233414 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:47.233469 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:47.233546 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:47.234361 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.234840 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:47.235531 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:47.235761 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:47.237907 (+ 2146us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:47.237926 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:47.237929 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:47.237954 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:47.239735 (+ 1781us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:47.240185 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:47.240188 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:47.240189 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:47.240233 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:47.240396 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:47.240398 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:47.240399 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:47.240542 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:47.240639 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:47.240827 (+ 188us) server_negotiation.cc:300] Negotiation successful
1025 08:01:47.240919 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20251025 08:01:47.242453 28865 tablet_service.cc:3012] Rejecting scan request for tablet a824a7c6c0de4a61b5b4aef7eeb381f0: Uninitialized: clean time has not yet been initialized
I20251025 08:01:47.244073 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 28822
I20251025 08:01:47.251196 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 28958
I20251025 08:01:47.257112 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 29094
I20251025 08:01:47.262877 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 28726
2025-10-25T08:01:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3253 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:01:47 dist-test-slave-lq8l krb5kdc[29264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:01:47 dist-test-slave-lq8l krb5kdc[29264](info): set up 2 sockets
Oct 25 08:01:47 dist-test-slave-lq8l krb5kdc[29264](info): commencing operation
krb5kdc: starting...
W20251025 08:01:49.232194 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:01:49 dist-test-slave-lq8l krb5kdc[29264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:01:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:49.367987 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:40031
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:49.475587 29280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:49.475826 29280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:49.475883 29280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:49.479136 29280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:01:49.479207 29280 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251025 08:01:49.479225 29280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:49.479250 29280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:49.479269 29280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:01:49.479300 29280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:01:49.483713 29280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:40031
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:40031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:49.484848 29280 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:49.485733 29280 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:49.491365 29288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:49.491369 29285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:49.491365 29286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:49.491730 29280 server_base.cc:1047] running on GCE node
I20251025 08:01:49.492262 29280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:49.493206 29280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:49.494376 29280 hybrid_clock.cc:648] HybridClock initialized: now 1761379309494351 us; error 45 us; skew 500 ppm
Oct 25 08:01:49 dist-test-slave-lq8l krb5kdc[29264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379309, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:49.497066 29280 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:01:49.498147 29280 webserver.cc:492] Webserver started at http://127.27.132.62:45259/ using document root <none> and password file <none>
I20251025 08:01:49.498687 29280 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:49.498736 29280 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:49.498903 29280 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:49.500694 29280 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "1d85f35b610340ef8e65d27c2a6f6db5"
format_stamp: "Formatted at 2025-10-25 08:01:49 on dist-test-slave-lq8l"
server_key: "48055a881608989dba3f7b3524650e68"
server_key_iv: "3ba8fdc7ee5a251369dc7f31d783df1a"
server_key_version: "encryptionkey@0"
I20251025 08:01:49.501147 29280 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "1d85f35b610340ef8e65d27c2a6f6db5"
format_stamp: "Formatted at 2025-10-25 08:01:49 on dist-test-slave-lq8l"
server_key: "48055a881608989dba3f7b3524650e68"
server_key_iv: "3ba8fdc7ee5a251369dc7f31d783df1a"
server_key_version: "encryptionkey@0"
I20251025 08:01:49.504578 29280 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:01:49.506793 29295 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:49.507987 29280 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:01:49.508157 29280 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "1d85f35b610340ef8e65d27c2a6f6db5"
format_stamp: "Formatted at 2025-10-25 08:01:49 on dist-test-slave-lq8l"
server_key: "48055a881608989dba3f7b3524650e68"
server_key_iv: "3ba8fdc7ee5a251369dc7f31d783df1a"
server_key_version: "encryptionkey@0"
I20251025 08:01:49.508273 29280 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:49.532958 29280 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:49.536234 29280 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:49.536404 29280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:49.544727 29280 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:40031
I20251025 08:01:49.545015 29357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:40031 every 8 connection(s)
I20251025 08:01:49.546082 29280 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:01:49.549044 29358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:49.554378 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 29280
I20251025 08:01:49.554507 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:01:49.554597 29358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5: Bootstrap starting.
I20251025 08:01:49.554754 28176 external_mini_cluster.cc:1468] Setting key 622f70a23c22b2b79015511f0e4f2442
I20251025 08:01:49.557265 29358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:49.558092 29358 log.cc:826] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:49.560318 29358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5: No bootstrap required, opened a new log
Oct 25 08:01:49 dist-test-slave-lq8l krb5kdc[29264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:49.564033 29358 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d85f35b610340ef8e65d27c2a6f6db5" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40031 } }
I20251025 08:01:49.564239 29358 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:49.564287 29358 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d85f35b610340ef8e65d27c2a6f6db5, State: Initialized, Role: FOLLOWER
I20251025 08:01:49.564764 29358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d85f35b610340ef8e65d27c2a6f6db5" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40031 } }
I20251025 08:01:49.564888 29358 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:01:49.564949 29358 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:01:49.565044 29358 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:49.565990 29358 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d85f35b610340ef8e65d27c2a6f6db5" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40031 } }
I20251025 08:01:49.566370 29358 leader_election.cc:304] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d85f35b610340ef8e65d27c2a6f6db5; no voters:
I20251025 08:01:49.566744 29358 leader_election.cc:290] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:01:49.566871 29363 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:49.567101 29363 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [term 1 LEADER]: Becoming Leader. State: Replica: 1d85f35b610340ef8e65d27c2a6f6db5, State: Running, Role: LEADER
I20251025 08:01:49.567550 29363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d85f35b610340ef8e65d27c2a6f6db5" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40031 } }
I20251025 08:01:49.567896 29358 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:01:49.568987 29363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d85f35b610340ef8e65d27c2a6f6db5. Latest consensus state: current_term: 1 leader_uuid: "1d85f35b610340ef8e65d27c2a6f6db5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d85f35b610340ef8e65d27c2a6f6db5" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40031 } } }
I20251025 08:01:49.569172 29363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [sys.catalog]: This master's current role is: LEADER
I20251025 08:01:49.569239 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:49.556068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.27.132.62:40031)
1025 08:01:49.556515 (+ 447us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:49.556525 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:49.556560 (+ 35us) server_negotiation.cc:408] Connection header received
1025 08:01:49.557211 (+ 651us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:49.557233 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:49.557594 (+ 361us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:49.557906 (+ 312us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:49.558952 (+ 1046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:49.559803 (+ 851us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:49.560650 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:49.560889 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:49.563512 (+ 2623us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:49.563532 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:49.563544 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:49.563582 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:49.565607 (+ 2025us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:49.566131 (+ 524us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:49.566137 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:49.566143 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:49.566230 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:49.566535 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:49.566542 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:49.566547 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:49.567070 (+ 523us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:49.567996 (+ 926us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:49.568173 (+ 177us) server_negotiation.cc:300] Negotiation successful
1025 08:01:49.568451 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":154,"threads_started":1}
I20251025 08:01:49.569344 29364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d85f35b610340ef8e65d27c2a6f6db5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d85f35b610340ef8e65d27c2a6f6db5" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40031 } } }
I20251025 08:01:49.569437 29364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5 [sys.catalog]: This master's current role is: LEADER
W20251025 08:01:49.572837 29378 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 08:01:49.572912 29378 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 08:01:49.573019 29372 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:01:49.573930 29372 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:01:49.579295 29372 catalog_manager.cc:1357] Generated new cluster ID: 8d7d5b04244b47db80023a9cb6e857c5
I20251025 08:01:49.579399 29372 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:01:49.607002 29372 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:01:49.608035 29372 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:01:49.616016 29372 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5: Generated new TSK 0
I20251025 08:01:49.616966 29372 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:49.675127 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:40031
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:49.787657 29386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:49.787904 29386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:49.788010 29386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:49.791316 29386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:49.791421 29386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:49.791518 29386 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 08:01:49.791599 29386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:01:49.795951 29386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.132.62:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:49.797096 29386 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:49.797981 29386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:49.804574 29391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:49.804585 29394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:49.804759 29386 server_base.cc:1047] running on GCE node
W20251025 08:01:49.804574 29392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:49.805155 29386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:49.805717 29386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:49.806882 29386 hybrid_clock.cc:648] HybridClock initialized: now 1761379309806861 us; error 36 us; skew 500 ppm
Oct 25 08:01:49 dist-test-slave-lq8l krb5kdc[29264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379309, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:49.809687 29386 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:01:49.810827 29386 webserver.cc:492] Webserver started at http://127.27.132.1:45525/ using document root <none> and password file <none>
I20251025 08:01:49.811429 29386 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:49.811483 29386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:49.811703 29386 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:49.813517 29386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "10130032710a4718b7d2ced43921f4a1"
format_stamp: "Formatted at 2025-10-25 08:01:49 on dist-test-slave-lq8l"
server_key: "a260e5ce47f74c9ed6b62c7957b308af"
server_key_iv: "63add2d16d919a72a4220d86ea9ec4d0"
server_key_version: "encryptionkey@0"
I20251025 08:01:49.814004 29386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "10130032710a4718b7d2ced43921f4a1"
format_stamp: "Formatted at 2025-10-25 08:01:49 on dist-test-slave-lq8l"
server_key: "a260e5ce47f74c9ed6b62c7957b308af"
server_key_iv: "63add2d16d919a72a4220d86ea9ec4d0"
server_key_version: "encryptionkey@0"
I20251025 08:01:49.817462 29386 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:01:49.819670 29401 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:49.820683 29386 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:01:49.820775 29386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "10130032710a4718b7d2ced43921f4a1"
format_stamp: "Formatted at 2025-10-25 08:01:49 on dist-test-slave-lq8l"
server_key: "a260e5ce47f74c9ed6b62c7957b308af"
server_key_iv: "63add2d16d919a72a4220d86ea9ec4d0"
server_key_version: "encryptionkey@0"
I20251025 08:01:49.820871 29386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:49.831624 29386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:49.834542 29386 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:49.834725 29386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:49.836246 29386 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:49.836308 29386 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:49.836361 29386 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:49.836392 29386 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 08:01:49 dist-test-slave-lq8l krb5kdc[29264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379309, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:49.848165 29386 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:34023
I20251025 08:01:49.848440 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:49.836810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:42535 (local address 127.27.132.62:40031)
1025 08:01:49.836947 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:49.836951 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:49.837724 (+ 773us) server_negotiation.cc:408] Connection header received
1025 08:01:49.838577 (+ 853us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:49.838581 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:49.838647 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:49.838726 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:49.840195 (+ 1469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:49.840837 (+ 642us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:49.841625 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:49.841874 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:49.844425 (+ 2551us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:49.844454 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:49.844459 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:49.844508 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:49.846246 (+ 1738us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:49.846958 (+ 712us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:49.846965 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:49.846970 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:49.847022 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:49.847283 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:49.847286 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:49.847288 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:49.847554 (+ 266us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:49.847662 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:49.848146 (+ 484us) server_negotiation.cc:300] Negotiation successful
1025 08:01:49.848279 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251025 08:01:49.849032 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:49.837149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:40031 (local address 127.27.132.1:42535)
1025 08:01:49.837593 (+ 444us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:49.837628 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:49.838380 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:49.838893 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:49.838900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:49.839418 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:49.840047 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:49.840060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:49.840966 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:49.840970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:49.841499 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:49.841508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:49.841742 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:49.842385 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:49.842432 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:49.844275 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:49.846437 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:49.846445 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:49.846458 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:49.846826 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:49.847111 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:49.847114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:49.847115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:49.847197 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:49.847651 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:49.847656 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:49.847882 (+ 226us) client_negotiation.cc:769] Sending connection context
1025 08:01:49.848097 (+ 215us) client_negotiation.cc:240] Negotiation successful
1025 08:01:49.848353 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
I20251025 08:01:49.849175 29386 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:01:49.849735 29516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:34023 every 8 connection(s)
I20251025 08:01:49.850495 29517 heartbeater.cc:344] Connected to a master server at 127.27.132.62:40031
I20251025 08:01:49.850725 29517 heartbeater.cc:461] Registering TS with master...
I20251025 08:01:49.851176 29517 heartbeater.cc:507] Master 127.27.132.62:40031 requested a full tablet report, sending...
I20251025 08:01:49.851442 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 29386
I20251025 08:01:49.851545 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:01:49.851823 28176 external_mini_cluster.cc:1468] Setting key 884acfe46ddd66b4fc9c06537d992285
I20251025 08:01:49.852738 29312 ts_manager.cc:194] Registered new tserver with Master: 10130032710a4718b7d2ced43921f4a1 (127.27.132.1:34023)
I20251025 08:01:49.854135 29312 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:42535
I20251025 08:01:49.859512 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:49.852654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:34837 (local address 127.27.132.62:40031)
1025 08:01:49.852772 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:49.852776 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:49.852977 (+ 201us) server_negotiation.cc:408] Connection header received
1025 08:01:49.853095 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:49.853099 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:49.853138 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:49.853213 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:49.853814 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:49.854515 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:49.855271 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:49.855510 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:49.856305 (+ 795us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:49.856321 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:49.856323 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:49.856343 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:49.858128 (+ 1785us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:49.858516 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:49.858519 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:49.858520 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:49.858561 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:49.858779 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:49.858782 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:49.858784 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:49.858909 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:49.858994 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:49.859218 (+ 224us) server_negotiation.cc:300] Negotiation successful
1025 08:01:49.859365 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251025 08:01:49.859555 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:49.852593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:40031 (local address 127.27.132.1:34837)
1025 08:01:49.852886 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:49.852896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:49.853006 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:49.853254 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:49.853256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:49.853418 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:49.853680 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:49.853697 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:49.854650 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:49.854656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:49.855166 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:49.855174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:49.855278 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:49.855913 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:49.855925 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:49.856190 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:49.858237 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:49.858240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:49.858241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:49.858413 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:49.858653 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:49.858656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:49.858658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:49.858702 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:49.858993 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:49.858996 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:49.859073 (+ 77us) client_negotiation.cc:769] Sending connection context
1025 08:01:49.859161 (+ 88us) client_negotiation.cc:240] Negotiation successful
1025 08:01:49.859326 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:49.898490 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:40031
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:50.007282 29528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:50.007546 29528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:50.007611 29528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:50.010917 29528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:50.011018 29528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:50.011075 29528 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 08:01:50.011152 29528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:01:50.015636 29528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.132.62:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:50.016871 29528 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:50.017742 29528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:50.024174 29533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:50.024170 29534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:50.024391 29528 server_base.cc:1047] running on GCE node
W20251025 08:01:50.024170 29536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:50.024874 29528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:50.025452 29528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:50.026613 29528 hybrid_clock.cc:648] HybridClock initialized: now 1761379310026590 us; error 39 us; skew 500 ppm
Oct 25 08:01:50 dist-test-slave-lq8l krb5kdc[29264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:50.029335 29528 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:01:50.030426 29528 webserver.cc:492] Webserver started at http://127.27.132.2:40511/ using document root <none> and password file <none>
I20251025 08:01:50.030972 29528 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:50.031020 29528 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:50.031203 29528 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:50.033149 29528 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "447c594df2cc412789f5ab82d92a0963"
format_stamp: "Formatted at 2025-10-25 08:01:50 on dist-test-slave-lq8l"
server_key: "56530ec5596632144d9bb14a68288e90"
server_key_iv: "007b11b74fddae7ab22c9361f0b8f0f3"
server_key_version: "encryptionkey@0"
I20251025 08:01:50.033672 29528 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "447c594df2cc412789f5ab82d92a0963"
format_stamp: "Formatted at 2025-10-25 08:01:50 on dist-test-slave-lq8l"
server_key: "56530ec5596632144d9bb14a68288e90"
server_key_iv: "007b11b74fddae7ab22c9361f0b8f0f3"
server_key_version: "encryptionkey@0"
I20251025 08:01:50.037228 29528 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:01:50.039546 29543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:50.040560 29528 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:01:50.040655 29528 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "447c594df2cc412789f5ab82d92a0963"
format_stamp: "Formatted at 2025-10-25 08:01:50 on dist-test-slave-lq8l"
server_key: "56530ec5596632144d9bb14a68288e90"
server_key_iv: "007b11b74fddae7ab22c9361f0b8f0f3"
server_key_version: "encryptionkey@0"
I20251025 08:01:50.040755 29528 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:50.054278 29528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:50.057518 29528 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:50.057690 29528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:50.059198 29528 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:50.059268 29528 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:50.059319 29528 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:50.059384 29528 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 08:01:50 dist-test-slave-lq8l krb5kdc[29264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:50.070835 29528 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:37087
I20251025 08:01:50.070856 29658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:37087 every 8 connection(s)
I20251025 08:01:50.071842 29528 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:01:50.072183 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.059952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:48055 (local address 127.27.132.62:40031)
1025 08:01:50.060105 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.060110 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.060876 (+ 766us) server_negotiation.cc:408] Connection header received
1025 08:01:50.061985 (+ 1109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.061989 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.062041 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.062140 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:50.063994 (+ 1854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.064641 (+ 647us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.065330 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.065527 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.067868 (+ 2341us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:50.067896 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:50.067901 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:50.067939 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:50.070077 (+ 2138us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.070730 (+ 653us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.070734 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.070736 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.070787 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.071101 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.071105 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.071107 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.071272 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:50.071393 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.071888 (+ 495us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.072016 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251025 08:01:50.072867 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.060197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:40031 (local address 127.27.132.2:48055)
1025 08:01:50.060713 (+ 516us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.060758 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.061730 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.062334 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.062347 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.062948 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:50.063810 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.063832 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.064781 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.064784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.065222 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.065234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.065530 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.066141 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:50.066174 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:50.067714 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:50.070224 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:50.070231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:50.070243 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:50.070584 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:50.070881 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:50.070884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:50.070886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:50.071002 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:50.071401 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:50.071411 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:50.071641 (+ 230us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.071833 (+ 192us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.072105 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20251025 08:01:50.074172 29659 heartbeater.cc:344] Connected to a master server at 127.27.132.62:40031
I20251025 08:01:50.074347 29659 heartbeater.cc:461] Registering TS with master...
I20251025 08:01:50.074370 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 29528
I20251025 08:01:50.074487 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:01:50.074749 28176 external_mini_cluster.cc:1468] Setting key 7c7924ef734c183e67b19b604202a4ba
I20251025 08:01:50.074889 29659 heartbeater.cc:507] Master 127.27.132.62:40031 requested a full tablet report, sending...
I20251025 08:01:50.075908 29312 ts_manager.cc:194] Registered new tserver with Master: 447c594df2cc412789f5ab82d92a0963 (127.27.132.2:37087)
I20251025 08:01:50.076541 29312 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:48055
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
I20251025 08:01:50.084811 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.076607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:43211 (local address 127.27.132.62:40031)
1025 08:01:50.076845 (+ 238us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.076849 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.078019 (+ 1170us) server_negotiation.cc:408] Connection header received
1025 08:01:50.078121 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.078127 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.078183 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.078281 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:50.078904 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.079766 (+ 862us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.080406 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.080627 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.081322 (+ 695us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:50.081345 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:50.081352 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:50.081397 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:50.083166 (+ 1769us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.083634 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.083639 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.083643 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.083700 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.084009 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.084014 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.084016 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.084189 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:50.084288 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.084537 (+ 249us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.084654 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20251025 08:01:50.084828 29665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.076674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:40031 (local address 127.27.132.2:43211)
1025 08:01:50.077906 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.077922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.078026 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.078299 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.078302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.078503 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:50.078774 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.078785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.079876 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.079879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.080256 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.080263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.080357 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.080903 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:50.080916 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:50.081194 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:50.083292 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:50.083295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:50.083297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:50.083534 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:50.083827 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:50.083830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:50.083831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:50.083883 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:50.084288 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:50.084292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:50.084376 (+ 84us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.084475 (+ 99us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.084627 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:50.122731 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:40031
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:50.230809 29670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:50.231038 29670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:50.231132 29670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:50.234369 29670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:50.234458 29670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:50.234547 29670 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 08:01:50.234625 29670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:01:50.238859 29670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.132.62:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:50.240031 29670 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:50.240895 29670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:50.247454 29676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:50.247464 29678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:50.247542 29670 server_base.cc:1047] running on GCE node
W20251025 08:01:50.247478 29675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:50.248039 29670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:50.248601 29670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:50.249744 29670 hybrid_clock.cc:648] HybridClock initialized: now 1761379310249726 us; error 35 us; skew 500 ppm
Oct 25 08:01:50 dist-test-slave-lq8l krb5kdc[29264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:50.252475 29670 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:01:50.253491 29670 webserver.cc:492] Webserver started at http://127.27.132.3:38681/ using document root <none> and password file <none>
I20251025 08:01:50.254016 29670 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:50.254062 29670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:50.254222 29670 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:50.255921 29670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b"
format_stamp: "Formatted at 2025-10-25 08:01:50 on dist-test-slave-lq8l"
server_key: "863c8dc259ba76a91bc4be74b4952d22"
server_key_iv: "52f0a24b368526eaa4a65b23754fec36"
server_key_version: "encryptionkey@0"
I20251025 08:01:50.256366 29670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b"
format_stamp: "Formatted at 2025-10-25 08:01:50 on dist-test-slave-lq8l"
server_key: "863c8dc259ba76a91bc4be74b4952d22"
server_key_iv: "52f0a24b368526eaa4a65b23754fec36"
server_key_version: "encryptionkey@0"
I20251025 08:01:50.259924 29670 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:01:50.262072 29685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:50.263089 29670 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:01:50.263190 29670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b"
format_stamp: "Formatted at 2025-10-25 08:01:50 on dist-test-slave-lq8l"
server_key: "863c8dc259ba76a91bc4be74b4952d22"
server_key_iv: "52f0a24b368526eaa4a65b23754fec36"
server_key_version: "encryptionkey@0"
I20251025 08:01:50.263270 29670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:50.276675 29670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:50.279840 29670 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:50.280004 29670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:50.281424 29670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:50.281477 29670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:50.281520 29670 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:50.281553 29670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 08:01:50 dist-test-slave-lq8l krb5kdc[29264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:50.292076 29670 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:34567
I20251025 08:01:50.292804 29800 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:34567 every 8 connection(s)
I20251025 08:01:50.293079 29670 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:01:50.294611 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.282181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:49939 (local address 127.27.132.62:40031)
1025 08:01:50.282346 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.282352 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.283025 (+ 673us) server_negotiation.cc:408] Connection header received
1025 08:01:50.284031 (+ 1006us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.284036 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.284097 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.284205 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:50.286026 (+ 1821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.286927 (+ 901us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.287969 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.288213 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.290898 (+ 2685us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:50.290924 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:50.290931 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:50.290969 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:50.292598 (+ 1629us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.293156 (+ 558us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.293161 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.293163 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.293219 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.293551 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.293554 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.293556 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.293760 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:50.293847 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.294376 (+ 529us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.294489 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251025 08:01:50.295203 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.282422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:40031 (local address 127.27.132.3:49939)
1025 08:01:50.282862 (+ 440us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.282905 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.283785 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.284389 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.284402 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.285017 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:50.285810 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.285831 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.287072 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.287078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.287773 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.287785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.288035 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.288775 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:50.288814 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:50.290727 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:50.292732 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:50.292739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:50.292752 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:50.293051 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:50.293317 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:50.293320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:50.293321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:50.293434 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:50.293855 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:50.293861 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:50.294101 (+ 240us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.294286 (+ 185us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.294511 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":128,"threads_started":1}
I20251025 08:01:50.296481 29801 heartbeater.cc:344] Connected to a master server at 127.27.132.62:40031
I20251025 08:01:50.296666 29801 heartbeater.cc:461] Registering TS with master...
I20251025 08:01:50.297169 29801 heartbeater.cc:507] Master 127.27.132.62:40031 requested a full tablet report, sending...
I20251025 08:01:50.298152 29311 ts_manager.cc:194] Registered new tserver with Master: c4c28a25ba7e4cb0b07c37f7c45a272b (127.27.132.3:34567)
I20251025 08:01:50.298743 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 29670
I20251025 08:01:50.298844 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:01:50.298918 29311 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:49939
I20251025 08:01:50.299125 28176 external_mini_cluster.cc:1468] Setting key ac16a7e873905c8331ee945e9ebf0708
I20251025 08:01:50.301191 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 08:01:50.305362 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.298497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:58439 (local address 127.27.132.62:40031)
1025 08:01:50.298622 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.298626 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.298833 (+ 207us) server_negotiation.cc:408] Connection header received
1025 08:01:50.298956 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.298959 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.299004 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.299070 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:50.299692 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.300368 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.301154 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.301331 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.302346 (+ 1015us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:50.302360 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:50.302362 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:50.302385 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:50.303855 (+ 1470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.304321 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.304323 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.304325 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.304361 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.304599 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.304601 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.304602 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.304728 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:50.304798 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.305137 (+ 339us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.305238 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 08:01:50.305413 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.298399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:40031 (local address 127.27.132.3:58439)
1025 08:01:50.298745 (+ 346us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.298756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.298859 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.299093 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.299096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.299282 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:50.299575 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.299583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.300482 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.300486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.300879 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.300890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.301004 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.301760 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:50.301780 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:50.302212 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:50.303976 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:50.303983 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:50.303988 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:50.304226 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:50.304444 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:50.304451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:50.304455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:50.304523 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:50.304806 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:50.304812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:50.304913 (+ 101us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.305032 (+ 119us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.305190 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":86,"threads_started":1}
Oct 25 08:01:50 dist-test-slave-lq8l krb5kdc[29264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 25 08:01:50 dist-test-slave-lq8l krb5kdc[29264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:50.326221 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.318562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.27.132.62:40031)
1025 08:01:50.318685 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.318688 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.318708 (+ 20us) server_negotiation.cc:408] Connection header received
1025 08:01:50.318749 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.318752 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.318796 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.318875 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:50.319794 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.320295 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.321097 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.321247 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.323435 (+ 2188us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:50.323454 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:50.323457 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:50.323484 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:50.324904 (+ 1420us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.325295 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.325298 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.325300 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.325344 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.325614 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.325615 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.325617 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.325763 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:50.325861 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.326005 (+ 144us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.326096 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251025 08:01:50.328912 29311 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:01:50.331243 29311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:01:50.344250 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.339963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:34567 (local address 127.0.0.1:59554)
1025 08:01:50.340428 (+ 465us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.340444 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.340531 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.341239 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.341243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.341279 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:50.341600 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.341606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.343092 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.343095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.343919 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.343926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.344021 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.344036 (+ 15us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.344081 (+ 45us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.344119 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"spinlock_wait_cycles":9472,"thread_start_us":98,"threads_started":1}
I20251025 08:01:50.344250 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.339219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:34023 (local address 127.0.0.1:49394)
1025 08:01:50.339868 (+ 649us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.339909 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.340050 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.341041 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.341051 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.341088 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:50.341370 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.341378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.342794 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.342797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.343761 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.343771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.343851 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.343910 (+ 59us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.343995 (+ 85us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.344095 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":135,"threads_started":1}
I20251025 08:01:50.345129 29820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.339499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:37087 (local address 127.0.0.1:39566)
1025 08:01:50.340166 (+ 667us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.340181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.340271 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.341239 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.341243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.341285 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:50.341592 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.341602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.342891 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.342897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.344020 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.344034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.344143 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.344161 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.344210 (+ 49us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.344267 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"spinlock_wait_cycles":10368,"thread_start_us":95,"threads_started":1}
I20251025 08:01:50.345258 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.339867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.27.132.2:37087)
1025 08:01:50.340720 (+ 853us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.340727 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.340748 (+ 21us) server_negotiation.cc:408] Connection header received
1025 08:01:50.340817 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.340822 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.340960 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.341081 (+ 121us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:50.341762 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.342664 (+ 902us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.344404 (+ 1740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.344907 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.344998 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.345054 (+ 56us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.345128 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":123,"threads_started":1}
I20251025 08:01:50.345276 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.339385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.27.132.1:34023)
1025 08:01:50.340528 (+ 1143us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.340536 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.340562 (+ 26us) server_negotiation.cc:408] Connection header received
1025 08:01:50.340655 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.340662 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.340875 (+ 213us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.341033 (+ 158us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:50.341505 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.342630 (+ 1125us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.344130 (+ 1500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.344690 (+ 560us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.344883 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.344954 (+ 71us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.345048 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":101,"threads_started":1}
I20251025 08:01:50.345611 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.340371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.27.132.3:34567)
1025 08:01:50.340767 (+ 396us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.340772 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.340791 (+ 19us) server_negotiation.cc:408] Connection header received
1025 08:01:50.340856 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.340861 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.341000 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.341138 (+ 138us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:50.341723 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.342972 (+ 1249us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.344398 (+ 1426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.345261 (+ 863us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.345364 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.345431 (+ 67us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.345507 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":111,"threads_started":1}
I20251025 08:01:50.348198 29593 tablet_service.cc:1505] Processing CreateTablet for tablet ac1e01ff16ea4c5dabaef62fcd564bd1 (DEFAULT_TABLE table=test-table [id=01223783ef4042fc9a62d5c41fec60f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:50.348207 29735 tablet_service.cc:1505] Processing CreateTablet for tablet ac1e01ff16ea4c5dabaef62fcd564bd1 (DEFAULT_TABLE table=test-table [id=01223783ef4042fc9a62d5c41fec60f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:50.348207 29446 tablet_service.cc:1505] Processing CreateTablet for tablet ac1e01ff16ea4c5dabaef62fcd564bd1 (DEFAULT_TABLE table=test-table [id=01223783ef4042fc9a62d5c41fec60f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:50.349085 29593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1e01ff16ea4c5dabaef62fcd564bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:50.349095 29446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1e01ff16ea4c5dabaef62fcd564bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:50.349095 29735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1e01ff16ea4c5dabaef62fcd564bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:50.354522 29824 tablet_bootstrap.cc:492] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: Bootstrap starting.
I20251025 08:01:50.355726 29825 tablet_bootstrap.cc:492] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1: Bootstrap starting.
I20251025 08:01:50.355726 29826 tablet_bootstrap.cc:492] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: Bootstrap starting.
I20251025 08:01:50.357317 29824 tablet_bootstrap.cc:654] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:50.357816 29826 tablet_bootstrap.cc:654] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:50.357816 29825 tablet_bootstrap.cc:654] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:50.358378 29824 log.cc:826] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:50.358680 29825 log.cc:826] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:50.359050 29826 log.cc:826] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:50.360500 29824 tablet_bootstrap.cc:492] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: No bootstrap required, opened a new log
I20251025 08:01:50.360500 29825 tablet_bootstrap.cc:492] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1: No bootstrap required, opened a new log
I20251025 08:01:50.360646 29826 tablet_bootstrap.cc:492] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: No bootstrap required, opened a new log
I20251025 08:01:50.360695 29824 ts_tablet_manager.cc:1403] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251025 08:01:50.360695 29825 ts_tablet_manager.cc:1403] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 08:01:50.360996 29826 ts_tablet_manager.cc:1403] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20251025 08:01:50.363606 29826 raft_consensus.cc:359] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.363803 29826 raft_consensus.cc:385] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:50.363888 29826 raft_consensus.cc:740] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c28a25ba7e4cb0b07c37f7c45a272b, State: Initialized, Role: FOLLOWER
I20251025 08:01:50.363871 29824 raft_consensus.cc:359] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.364079 29824 raft_consensus.cc:385] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:50.364193 29824 raft_consensus.cc:740] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447c594df2cc412789f5ab82d92a0963, State: Initialized, Role: FOLLOWER
I20251025 08:01:50.364315 29826 consensus_queue.cc:260] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.364262 29825 raft_consensus.cc:359] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.364492 29825 raft_consensus.cc:385] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:50.364557 29825 raft_consensus.cc:740] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10130032710a4718b7d2ced43921f4a1, State: Initialized, Role: FOLLOWER
I20251025 08:01:50.364683 29824 consensus_queue.cc:260] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.364996 29801 heartbeater.cc:499] Master 127.27.132.62:40031 was elected leader, sending a full tablet report...
I20251025 08:01:50.364989 29825 consensus_queue.cc:260] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.365195 29826 ts_tablet_manager.cc:1434] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251025 08:01:50.365571 29824 ts_tablet_manager.cc:1434] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251025 08:01:50.365646 29825 ts_tablet_manager.cc:1434] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251025 08:01:50.365923 29659 heartbeater.cc:499] Master 127.27.132.62:40031 was elected leader, sending a full tablet report...
I20251025 08:01:50.365958 29517 heartbeater.cc:499] Master 127.27.132.62:40031 was elected leader, sending a full tablet report...
I20251025 08:01:50.529805 29830 raft_consensus.cc:493] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:01:50.530007 29830 raft_consensus.cc:515] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.531153 29830 leader_election.cc:290] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10130032710a4718b7d2ced43921f4a1 (127.27.132.1:34023), 447c594df2cc412789f5ab82d92a0963 (127.27.132.2:37087)
I20251025 08:01:50.533377 29832 raft_consensus.cc:493] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:01:50.533587 29832 raft_consensus.cc:515] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.534606 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.531505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:34023 (local address 127.27.132.3:45527)
1025 08:01:50.531873 (+ 368us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.531887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.531970 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.532221 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.532224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.532237 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:50.532507 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.532515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.533527 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.533531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.534196 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.534205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.534321 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.534343 (+ 22us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.534397 (+ 54us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.534455 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":117,"threads_started":1}
I20251025 08:01:50.534746 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.531459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:37087 (local address 127.27.132.3:35283)
1025 08:01:50.531675 (+ 216us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.531691 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.531800 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.532095 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.532097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.532110 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:50.532378 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.532385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.533491 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.533497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.534347 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.534357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.534498 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.534515 (+ 17us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.534564 (+ 49us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.534630 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20251025 08:01:50.535096 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.531550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:45527 (local address 127.27.132.1:34023)
1025 08:01:50.531693 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.531697 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.531967 (+ 270us) server_negotiation.cc:408] Connection header received
1025 08:01:50.532064 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.532069 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.532118 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.532194 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:50.532639 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.533375 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.534324 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.534830 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.534868 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.534918 (+ 50us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.534990 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 08:01:50.535260 29832 leader_election.cc:290] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 447c594df2cc412789f5ab82d92a0963 (127.27.132.2:37087), c4c28a25ba7e4cb0b07c37f7c45a272b (127.27.132.3:34567)
I20251025 08:01:50.535677 29471 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1e01ff16ea4c5dabaef62fcd564bd1" candidate_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10130032710a4718b7d2ced43921f4a1" is_pre_election: true
I20251025 08:01:50.535883 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.531550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:35283 (local address 127.27.132.2:37087)
1025 08:01:50.531693 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.531698 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.531773 (+ 75us) server_negotiation.cc:408] Connection header received
1025 08:01:50.531898 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.531900 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.531973 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.532078 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:50.532528 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.533348 (+ 820us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.535079 (+ 1731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.535609 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.535647 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.535706 (+ 59us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.535772 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251025 08:01:50.535872 29471 raft_consensus.cc:2468] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c28a25ba7e4cb0b07c37f7c45a272b in term 0.
I20251025 08:01:50.536319 29689 leader_election.cc:304] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10130032710a4718b7d2ced43921f4a1, c4c28a25ba7e4cb0b07c37f7c45a272b; no voters:
I20251025 08:01:50.536703 29830 raft_consensus.cc:2804] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:01:50.536857 29830 raft_consensus.cc:493] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:01:50.536978 29830 raft_consensus.cc:3060] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:50.536974 29613 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1e01ff16ea4c5dabaef62fcd564bd1" candidate_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447c594df2cc412789f5ab82d92a0963" is_pre_election: true
I20251025 08:01:50.537202 29613 raft_consensus.cc:2468] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c28a25ba7e4cb0b07c37f7c45a272b in term 0.
I20251025 08:01:50.538300 29830 raft_consensus.cc:515] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.538411 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.534946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:37087 (local address 127.27.132.1:39259)
1025 08:01:50.535724 (+ 778us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.535739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.535826 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.536185 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.536187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.536204 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:50.536399 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.536404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.537324 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.537327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.538041 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.538050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.538159 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.538177 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.538228 (+ 51us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.538280 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":84,"threads_started":1}
I20251025 08:01:50.539077 29830 leader_election.cc:290] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [CANDIDATE]: Term 1 election: Requested vote from peers 10130032710a4718b7d2ced43921f4a1 (127.27.132.1:34023), 447c594df2cc412789f5ab82d92a0963 (127.27.132.2:37087)
I20251025 08:01:50.539186 29613 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1e01ff16ea4c5dabaef62fcd564bd1" candidate_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447c594df2cc412789f5ab82d92a0963"
I20251025 08:01:50.539258 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.535229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:39259 (local address 127.27.132.2:37087)
1025 08:01:50.535888 (+ 659us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.535894 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.535908 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:01:50.535947 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.535951 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.536002 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.536081 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:50.536532 (+ 451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.537193 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.538159 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.538660 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.538697 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.538744 (+ 47us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.538804 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":131,"threads_started":1}
I20251025 08:01:50.539311 29613 raft_consensus.cc:3060] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:50.539656 29612 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1e01ff16ea4c5dabaef62fcd564bd1" candidate_uuid: "10130032710a4718b7d2ced43921f4a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447c594df2cc412789f5ab82d92a0963" is_pre_election: true
I20251025 08:01:50.539870 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.536154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:34567 (local address 127.27.132.1:48567)
1025 08:01:50.536872 (+ 718us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.536888 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.537018 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.537367 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.537371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.537390 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:50.537680 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.537690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.538641 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.538644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.539486 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.539495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.539609 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.539636 (+ 27us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.539695 (+ 59us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.539750 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":111,"threads_started":1}
I20251025 08:01:50.540323 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.536509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:48567 (local address 127.27.132.3:34567)
1025 08:01:50.536644 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.536648 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.536950 (+ 302us) server_negotiation.cc:408] Connection header received
1025 08:01:50.537138 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.537141 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.537189 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.537236 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:50.537817 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.538490 (+ 673us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.539619 (+ 1129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.540069 (+ 450us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.540100 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.540149 (+ 49us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.540209 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251025 08:01:50.540417 29471 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1e01ff16ea4c5dabaef62fcd564bd1" candidate_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10130032710a4718b7d2ced43921f4a1"
I20251025 08:01:50.540499 29613 raft_consensus.cc:2468] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c28a25ba7e4cb0b07c37f7c45a272b in term 1.
I20251025 08:01:50.540524 29471 raft_consensus.cc:3060] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:50.540834 29755 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1e01ff16ea4c5dabaef62fcd564bd1" candidate_uuid: "10130032710a4718b7d2ced43921f4a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" is_pre_election: true
I20251025 08:01:50.541035 29755 raft_consensus.cc:2393] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10130032710a4718b7d2ced43921f4a1 in current term 1: Already voted for candidate c4c28a25ba7e4cb0b07c37f7c45a272b in this term.
I20251025 08:01:50.541182 29686 leader_election.cc:304] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447c594df2cc412789f5ab82d92a0963, c4c28a25ba7e4cb0b07c37f7c45a272b; no voters:
I20251025 08:01:50.541455 29830 raft_consensus.cc:2804] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:50.541465 29405 leader_election.cc:304] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10130032710a4718b7d2ced43921f4a1; no voters: 447c594df2cc412789f5ab82d92a0963, c4c28a25ba7e4cb0b07c37f7c45a272b
I20251025 08:01:50.541680 29830 raft_consensus.cc:697] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 LEADER]: Becoming Leader. State: Replica: c4c28a25ba7e4cb0b07c37f7c45a272b, State: Running, Role: LEADER
I20251025 08:01:50.541675 29471 raft_consensus.cc:2468] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c28a25ba7e4cb0b07c37f7c45a272b in term 1.
I20251025 08:01:50.541752 29832 raft_consensus.cc:2749] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 08:01:50.542007 29830 consensus_queue.cc:237] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
W20251025 08:01:50.543928 29802 tablet.cc:2378] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 08:01:50.545107 29309 catalog_manager.cc:5649] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b reported cstate change: term changed from 0 to 1, leader changed from <none> to c4c28a25ba7e4cb0b07c37f7c45a272b (127.27.132.3). New cstate: current_term: 1 leader_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } health_report { overall_health: HEALTHY } } }
W20251025 08:01:50.573285 29660 tablet.cc:2378] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 08:01:50.582186 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.579452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.27.132.3:34567)
1025 08:01:50.579583 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.579587 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.579600 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:01:50.579701 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.579703 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.579757 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.579835 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:50.580157 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.580659 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.581260 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.581388 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.581439 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:50.581880 (+ 441us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:50.581969 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.582034 (+ 65us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.582081 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 08:01:50.587244 29613 raft_consensus.cc:1275] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963 [term 1 FOLLOWER]: Refusing update from remote peer c4c28a25ba7e4cb0b07c37f7c45a272b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:50.587282 29471 raft_consensus.cc:1275] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 1 FOLLOWER]: Refusing update from remote peer c4c28a25ba7e4cb0b07c37f7c45a272b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:50.587831 29837 consensus_queue.cc:1048] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [LEADER]: Connected to new peer: Peer: permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:50.587991 29830 consensus_queue.cc:1048] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:50.605979 29311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35900:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:50.606155 29311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35900:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:50.608639 29311 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5: Sending DeleteTablet for 3 replicas of tablet ac1e01ff16ea4c5dabaef62fcd564bd1
I20251025 08:01:50.609447 29593 tablet_service.cc:1552] Processing DeleteTablet for tablet ac1e01ff16ea4c5dabaef62fcd564bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:50 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:39566
I20251025 08:01:50.609459 29735 tablet_service.cc:1552] Processing DeleteTablet for tablet ac1e01ff16ea4c5dabaef62fcd564bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:50 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:59554
I20251025 08:01:50.609462 29446 tablet_service.cc:1552] Processing DeleteTablet for tablet ac1e01ff16ea4c5dabaef62fcd564bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:50 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:49394
I20251025 08:01:50.609951 29850 tablet_replica.cc:333] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: stopping tablet replica
I20251025 08:01:50.609962 29851 tablet_replica.cc:333] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: stopping tablet replica
I20251025 08:01:50.610220 29851 raft_consensus.cc:2243] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:01:50.610258 29850 raft_consensus.cc:2243] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:01:50.610689 29850 raft_consensus.cc:2272] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:50.610659 29311 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 08:01:50.610747 29851 raft_consensus.cc:2272] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 08:01:50.611107 29311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:01:50.611133 29852 tablet_replica.cc:333] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1: stopping tablet replica
I20251025 08:01:50.611395 29852 raft_consensus.cc:2243] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:01:50.611688 29852 raft_consensus.cc:2272] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:50.612838 29851 ts_tablet_manager.cc:1916] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:50.613235 29850 ts_tablet_manager.cc:1916] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:50.615631 29851 ts_tablet_manager.cc:1929] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:01:50.615768 29851 log.cc:1199] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/ac1e01ff16ea4c5dabaef62fcd564bd1
I20251025 08:01:50.616029 29850 ts_tablet_manager.cc:1929] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:01:50.616123 29851 ts_tablet_manager.cc:1950] T ac1e01ff16ea4c5dabaef62fcd564bd1 P c4c28a25ba7e4cb0b07c37f7c45a272b: Deleting consensus metadata
I20251025 08:01:50.616214 29850 log.cc:1199] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/ac1e01ff16ea4c5dabaef62fcd564bd1
I20251025 08:01:50.616626 29850 ts_tablet_manager.cc:1950] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 447c594df2cc412789f5ab82d92a0963: Deleting consensus metadata
I20251025 08:01:50.617674 29299 catalog_manager.cc:4985] TS c4c28a25ba7e4cb0b07c37f7c45a272b (127.27.132.3:34567): tablet ac1e01ff16ea4c5dabaef62fcd564bd1 (table test-table [id=01223783ef4042fc9a62d5c41fec60f1]) successfully deleted
I20251025 08:01:50.617671 29593 tablet_service.cc:1505] Processing CreateTablet for tablet 6e84b15ba9b84ade9bc7caadf68e936a (DEFAULT_TABLE table=test-table [id=4b761ac37e454a55aff589850ce3e3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:50.617959 29593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e84b15ba9b84ade9bc7caadf68e936a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:50.618054 29735 tablet_service.cc:1505] Processing CreateTablet for tablet 6e84b15ba9b84ade9bc7caadf68e936a (DEFAULT_TABLE table=test-table [id=4b761ac37e454a55aff589850ce3e3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:50.618136 29296 catalog_manager.cc:4985] TS 447c594df2cc412789f5ab82d92a0963 (127.27.132.2:37087): tablet ac1e01ff16ea4c5dabaef62fcd564bd1 (table test-table [id=01223783ef4042fc9a62d5c41fec60f1]) successfully deleted
I20251025 08:01:50.618291 29735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e84b15ba9b84ade9bc7caadf68e936a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:50.618292 29446 tablet_service.cc:1505] Processing CreateTablet for tablet 6e84b15ba9b84ade9bc7caadf68e936a (DEFAULT_TABLE table=test-table [id=4b761ac37e454a55aff589850ce3e3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:50.618551 29446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e84b15ba9b84ade9bc7caadf68e936a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:50.620424 29852 ts_tablet_manager.cc:1916] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:50.620447 29824 tablet_bootstrap.cc:492] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963: Bootstrap starting.
I20251025 08:01:50.620838 29825 tablet_bootstrap.cc:492] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1: Bootstrap starting.
I20251025 08:01:50.621454 29824 tablet_bootstrap.cc:654] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:50.621835 29825 tablet_bootstrap.cc:654] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:50.622121 29826 tablet_bootstrap.cc:492] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b: Bootstrap starting.
I20251025 08:01:50.622912 29852 ts_tablet_manager.cc:1929] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:01:50.622995 29852 log.cc:1199] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/wal/wals/ac1e01ff16ea4c5dabaef62fcd564bd1
I20251025 08:01:50.623075 29824 tablet_bootstrap.cc:492] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963: No bootstrap required, opened a new log
I20251025 08:01:50.623167 29824 ts_tablet_manager.cc:1403] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251025 08:01:50.623181 29826 tablet_bootstrap.cc:654] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:50.623782 29824 raft_consensus.cc:359] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.623908 29824 raft_consensus.cc:385] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:50.623948 29824 raft_consensus.cc:740] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447c594df2cc412789f5ab82d92a0963, State: Initialized, Role: FOLLOWER
I20251025 08:01:50.624104 29824 consensus_queue.cc:260] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.624200 29852 ts_tablet_manager.cc:1950] T ac1e01ff16ea4c5dabaef62fcd564bd1 P 10130032710a4718b7d2ced43921f4a1: Deleting consensus metadata
I20251025 08:01:50.624377 29826 tablet_bootstrap.cc:492] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b: No bootstrap required, opened a new log
I20251025 08:01:50.624372 29824 ts_tablet_manager.cc:1434] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:01:50.624459 29826 ts_tablet_manager.cc:1403] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20251025 08:01:50.624972 29826 raft_consensus.cc:359] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.625079 29826 raft_consensus.cc:385] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:50.625118 29826 raft_consensus.cc:740] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c28a25ba7e4cb0b07c37f7c45a272b, State: Initialized, Role: FOLLOWER
I20251025 08:01:50.625232 29826 consensus_queue.cc:260] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.625277 29299 catalog_manager.cc:4985] TS 10130032710a4718b7d2ced43921f4a1 (127.27.132.1:34023): tablet ac1e01ff16ea4c5dabaef62fcd564bd1 (table test-table [id=01223783ef4042fc9a62d5c41fec60f1]) successfully deleted
I20251025 08:01:50.625473 29826 ts_tablet_manager.cc:1434] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251025 08:01:50.626165 29825 tablet_bootstrap.cc:492] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1: No bootstrap required, opened a new log
I20251025 08:01:50.626255 29825 ts_tablet_manager.cc:1403] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20251025 08:01:50.626734 29825 raft_consensus.cc:359] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.626834 29825 raft_consensus.cc:385] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:50.626878 29825 raft_consensus.cc:740] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10130032710a4718b7d2ced43921f4a1, State: Initialized, Role: FOLLOWER
I20251025 08:01:50.626989 29825 consensus_queue.cc:260] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.627236 29825 ts_tablet_manager.cc:1434] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:01:50.631405 29837 raft_consensus.cc:493] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:01:50.631496 29837 raft_consensus.cc:515] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.631872 29837 leader_election.cc:290] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10130032710a4718b7d2ced43921f4a1 (127.27.132.1:34023), 447c594df2cc412789f5ab82d92a0963 (127.27.132.2:37087)
I20251025 08:01:50.632215 29471 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e84b15ba9b84ade9bc7caadf68e936a" candidate_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10130032710a4718b7d2ced43921f4a1" is_pre_election: true
I20251025 08:01:50.632333 29471 raft_consensus.cc:2468] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c28a25ba7e4cb0b07c37f7c45a272b in term 0.
I20251025 08:01:50.632557 29613 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e84b15ba9b84ade9bc7caadf68e936a" candidate_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447c594df2cc412789f5ab82d92a0963" is_pre_election: true
I20251025 08:01:50.632725 29613 raft_consensus.cc:2468] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c28a25ba7e4cb0b07c37f7c45a272b in term 0.
I20251025 08:01:50.632702 29689 leader_election.cc:304] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10130032710a4718b7d2ced43921f4a1, c4c28a25ba7e4cb0b07c37f7c45a272b; no voters:
I20251025 08:01:50.632876 29837 raft_consensus.cc:2804] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:01:50.632936 29837 raft_consensus.cc:493] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:01:50.632988 29837 raft_consensus.cc:3060] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:50.633854 29837 raft_consensus.cc:515] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.634138 29837 leader_election.cc:290] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [CANDIDATE]: Term 1 election: Requested vote from peers 10130032710a4718b7d2ced43921f4a1 (127.27.132.1:34023), 447c594df2cc412789f5ab82d92a0963 (127.27.132.2:37087)
I20251025 08:01:50.634421 29471 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e84b15ba9b84ade9bc7caadf68e936a" candidate_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10130032710a4718b7d2ced43921f4a1"
I20251025 08:01:50.634485 29613 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e84b15ba9b84ade9bc7caadf68e936a" candidate_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447c594df2cc412789f5ab82d92a0963"
I20251025 08:01:50.634526 29471 raft_consensus.cc:3060] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:50.634584 29613 raft_consensus.cc:3060] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:50.635736 29471 raft_consensus.cc:2468] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c28a25ba7e4cb0b07c37f7c45a272b in term 1.
I20251025 08:01:50.636046 29689 leader_election.cc:304] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10130032710a4718b7d2ced43921f4a1, c4c28a25ba7e4cb0b07c37f7c45a272b; no voters:
I20251025 08:01:50.636124 29613 raft_consensus.cc:2468] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c28a25ba7e4cb0b07c37f7c45a272b in term 1.
I20251025 08:01:50.636205 29837 raft_consensus.cc:2804] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:50.636281 29837 raft_consensus.cc:697] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 LEADER]: Becoming Leader. State: Replica: c4c28a25ba7e4cb0b07c37f7c45a272b, State: Running, Role: LEADER
I20251025 08:01:50.636395 29837 consensus_queue.cc:237] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.638007 29311 catalog_manager.cc:5649] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b reported cstate change: term changed from 0 to 1, leader changed from <none> to c4c28a25ba7e4cb0b07c37f7c45a272b (127.27.132.3). New cstate: current_term: 1 leader_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } health_report { overall_health: HEALTHY } } }
Oct 25 08:01:50 dist-test-slave-lq8l krb5kdc[29264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379309, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:50.681623 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.673243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.27.132.62:40031)
1025 08:01:50.673386 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.673391 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.673530 (+ 139us) server_negotiation.cc:408] Connection header received
1025 08:01:50.673693 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.673697 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.673752 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.673850 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:50.674660 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.675467 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.676092 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.676285 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.678412 (+ 2127us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:50.678430 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:50.678433 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:50.678456 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:50.679837 (+ 1381us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.680360 (+ 523us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.680382 (+ 22us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.680384 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.680427 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.680708 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.680712 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.680713 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.680870 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:50.680962 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.681276 (+ 314us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.681481 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":48}
I20251025 08:01:50.681633 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.673156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:40031 (local address 127.0.0.1:35916)
1025 08:01:50.673439 (+ 283us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.673456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.673582 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.673866 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.673868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.674308 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:50.674536 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.674543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.675575 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.675578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.675984 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.675992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.676096 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.676692 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:50.676706 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:50.678267 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:50.679943 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:50.679953 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:50.679959 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:50.680249 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:50.680521 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:50.680524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:50.680527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:50.680586 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:50.680982 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:50.680990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:50.681116 (+ 126us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.681238 (+ 122us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.681392 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
I20251025 08:01:50.685612 29311 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:35916:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251025 08:01:50.686393 29311 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:01:50.691282 29446 tablet_service.cc:1505] Processing CreateTablet for tablet cd34e9d70acb4c57bc206a65767e583d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58e35d4884f947228ce1449f5dd83641]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 08:01:50.691509 29593 tablet_service.cc:1505] Processing CreateTablet for tablet cd34e9d70acb4c57bc206a65767e583d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58e35d4884f947228ce1449f5dd83641]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 08:01:50.691712 29446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd34e9d70acb4c57bc206a65767e583d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:50.691736 29735 tablet_service.cc:1505] Processing CreateTablet for tablet cd34e9d70acb4c57bc206a65767e583d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58e35d4884f947228ce1449f5dd83641]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 08:01:50.691869 29593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd34e9d70acb4c57bc206a65767e583d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:50.692052 29735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd34e9d70acb4c57bc206a65767e583d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:50.694450 29825 tablet_bootstrap.cc:492] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1: Bootstrap starting.
I20251025 08:01:50.694478 29826 tablet_bootstrap.cc:492] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b: Bootstrap starting.
I20251025 08:01:50.695572 29825 tablet_bootstrap.cc:654] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:50.695676 29826 tablet_bootstrap.cc:654] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:50.696295 29824 tablet_bootstrap.cc:492] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: Bootstrap starting.
I20251025 08:01:50.697126 29826 tablet_bootstrap.cc:492] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b: No bootstrap required, opened a new log
I20251025 08:01:50.697216 29826 ts_tablet_manager.cc:1403] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251025 08:01:50.697356 29824 tablet_bootstrap.cc:654] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:50.697715 29826 raft_consensus.cc:359] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.697830 29826 raft_consensus.cc:385] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:50.697867 29826 raft_consensus.cc:740] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c28a25ba7e4cb0b07c37f7c45a272b, State: Initialized, Role: FOLLOWER
I20251025 08:01:50.698374 29826 consensus_queue.cc:260] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.698627 29837 tablet_replica.cc:442] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } } }
I20251025 08:01:50.698724 29837 tablet_replica.cc:445] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:50.698823 29824 tablet_bootstrap.cc:492] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: No bootstrap required, opened a new log
I20251025 08:01:50.698889 29824 ts_tablet_manager.cc:1403] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251025 08:01:50.699402 29824 raft_consensus.cc:359] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.699507 29824 raft_consensus.cc:385] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:50.699539 29824 raft_consensus.cc:740] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447c594df2cc412789f5ab82d92a0963, State: Initialized, Role: FOLLOWER
I20251025 08:01:50.699662 29824 consensus_queue.cc:260] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.699925 29824 ts_tablet_manager.cc:1434] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:01:50.700167 29831 tablet_replica.cc:442] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } } }
I20251025 08:01:50.700284 29831 tablet_replica.cc:445] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:50.700721 29826 ts_tablet_manager.cc:1434] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 08:01:50.701886 29825 tablet_bootstrap.cc:492] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1: No bootstrap required, opened a new log
I20251025 08:01:50.701974 29825 ts_tablet_manager.cc:1403] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20251025 08:01:50.702419 29825 raft_consensus.cc:359] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.702507 29825 raft_consensus.cc:385] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:50.702528 29825 raft_consensus.cc:740] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10130032710a4718b7d2ced43921f4a1, State: Initialized, Role: FOLLOWER
I20251025 08:01:50.702634 29825 consensus_queue.cc:260] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.702916 29825 ts_tablet_manager.cc:1434] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:01:50.702848 29832 tablet_replica.cc:442] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } } }
I20251025 08:01:50.702980 29832 tablet_replica.cc:445] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:50.736584 29831 raft_consensus.cc:493] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:01:50.736758 29831 raft_consensus.cc:515] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.738015 29831 leader_election.cc:290] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10130032710a4718b7d2ced43921f4a1 (127.27.132.1:34023), c4c28a25ba7e4cb0b07c37f7c45a272b (127.27.132.3:34567)
I20251025 08:01:50.741678 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.738680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:34567 (local address 127.27.132.2:38955)
1025 08:01:50.738966 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.738980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.739065 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.739373 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.739376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.739389 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:50.739655 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.739661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.740337 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.740340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.741233 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.741241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.741378 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.741415 (+ 37us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.741470 (+ 55us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.741530 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
I20251025 08:01:50.742139 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.738814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:38955 (local address 127.27.132.3:34567)
1025 08:01:50.738977 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.738983 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.739034 (+ 51us) server_negotiation.cc:408] Connection header received
1025 08:01:50.739171 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.739176 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.739239 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.739356 (+ 117us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:50.739763 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.740229 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.741373 (+ 1144us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.741867 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.741906 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.741957 (+ 51us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.742014 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251025 08:01:50.742507 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.738270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:43203 (local address 127.27.132.1:34023)
1025 08:01:50.738460 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.738465 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.738645 (+ 180us) server_negotiation.cc:408] Connection header received
1025 08:01:50.738885 (+ 240us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.738888 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.738948 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.739006 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:50.739598 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.740386 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.741425 (+ 1039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.741929 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.741965 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.742315 (+ 350us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.742387 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20251025 08:01:50.742563 29755 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd34e9d70acb4c57bc206a65767e583d" candidate_uuid: "447c594df2cc412789f5ab82d92a0963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" is_pre_election: true
I20251025 08:01:50.742717 29755 raft_consensus.cc:2468] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 447c594df2cc412789f5ab82d92a0963 in term 0.
I20251025 08:01:50.742937 29866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.738198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:34023 (local address 127.27.132.2:43203)
1025 08:01:50.738563 (+ 365us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.738580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.738740 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.739152 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.739155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.739172 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:50.739464 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.739474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.740497 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.740500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.741296 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.741306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.742156 (+ 850us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.742181 (+ 25us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.742698 (+ 517us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.742767 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":167,"threads_started":1}
I20251025 08:01:50.743081 29547 leader_election.cc:304] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447c594df2cc412789f5ab82d92a0963, c4c28a25ba7e4cb0b07c37f7c45a272b; no voters:
I20251025 08:01:50.743335 29831 raft_consensus.cc:2804] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:01:50.743431 29831 raft_consensus.cc:493] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:01:50.743474 29831 raft_consensus.cc:3060] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:50.743758 29471 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd34e9d70acb4c57bc206a65767e583d" candidate_uuid: "447c594df2cc412789f5ab82d92a0963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10130032710a4718b7d2ced43921f4a1" is_pre_election: true
I20251025 08:01:50.743947 29471 raft_consensus.cc:2468] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 447c594df2cc412789f5ab82d92a0963 in term 0.
I20251025 08:01:50.744630 29831 raft_consensus.cc:515] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.744956 29831 leader_election.cc:290] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [CANDIDATE]: Term 1 election: Requested vote from peers 10130032710a4718b7d2ced43921f4a1 (127.27.132.1:34023), c4c28a25ba7e4cb0b07c37f7c45a272b (127.27.132.3:34567)
I20251025 08:01:50.745316 29471 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd34e9d70acb4c57bc206a65767e583d" candidate_uuid: "447c594df2cc412789f5ab82d92a0963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10130032710a4718b7d2ced43921f4a1"
I20251025 08:01:50.745349 29755 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd34e9d70acb4c57bc206a65767e583d" candidate_uuid: "447c594df2cc412789f5ab82d92a0963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b"
I20251025 08:01:50.745445 29471 raft_consensus.cc:3060] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:50.745463 29755 raft_consensus.cc:3060] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:50.746347 29755 raft_consensus.cc:2468] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 447c594df2cc412789f5ab82d92a0963 in term 1.
I20251025 08:01:50.746646 29547 leader_election.cc:304] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447c594df2cc412789f5ab82d92a0963, c4c28a25ba7e4cb0b07c37f7c45a272b; no voters:
I20251025 08:01:50.746608 29471 raft_consensus.cc:2468] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 447c594df2cc412789f5ab82d92a0963 in term 1.
I20251025 08:01:50.746817 29831 raft_consensus.cc:2804] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:50.747035 29831 raft_consensus.cc:697] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [term 1 LEADER]: Becoming Leader. State: Replica: 447c594df2cc412789f5ab82d92a0963, State: Running, Role: LEADER
I20251025 08:01:50.747543 29831 consensus_queue.cc:237] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } }
I20251025 08:01:50.749392 29868 tablet_replica.cc:442] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: TxnStatusTablet state changed. Reason: New leader 447c594df2cc412789f5ab82d92a0963. Latest consensus state: current_term: 1 leader_uuid: "447c594df2cc412789f5ab82d92a0963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } } }
I20251025 08:01:50.749508 29868 tablet_replica.cc:445] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: This TxnStatusTablet replica's current role is: LEADER
I20251025 08:01:50.749775 29870 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251025 08:01:50.749898 29870 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251025 08:01:50.750149 29311 catalog_manager.cc:5649] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 reported cstate change: term changed from 0 to 1, leader changed from <none> to 447c594df2cc412789f5ab82d92a0963 (127.27.132.2). New cstate: current_term: 1 leader_uuid: "447c594df2cc412789f5ab82d92a0963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } health_report { overall_health: UNKNOWN } } }
I20251025 08:01:50.762010 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.758784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.27.132.2:37087)
1025 08:01:50.758914 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.758919 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.758937 (+ 18us) server_negotiation.cc:408] Connection header received
1025 08:01:50.759008 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.759012 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.759078 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.759192 (+ 114us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:50.759764 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.760250 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.760908 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.761102 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.761152 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:50.761536 (+ 384us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:50.761673 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.761829 (+ 156us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.761874 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251025 08:01:50.762010 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.758553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:37087 (local address 127.0.0.1:39570)
1025 08:01:50.758698 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.758715 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.758833 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.759195 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.759355 (+ 160us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.759368 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:01:50.759630 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.759641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.760367 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.760370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.760796 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.760804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.760907 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.760953 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:01:50.761724 (+ 771us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:01:50.761740 (+ 16us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.761806 (+ 66us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.761885 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20251025 08:01:50.765045 29755 raft_consensus.cc:1275] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 FOLLOWER]: Refusing update from remote peer 447c594df2cc412789f5ab82d92a0963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:50.765491 29471 raft_consensus.cc:1275] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1 [term 1 FOLLOWER]: Refusing update from remote peer 447c594df2cc412789f5ab82d92a0963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:50.765836 29831 consensus_queue.cc:1048] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:50.766103 29868 consensus_queue.cc:1048] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:50.767750 29832 tablet_replica.cc:442] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1: TxnStatusTablet state changed. Reason: New leader 447c594df2cc412789f5ab82d92a0963. Latest consensus state: current_term: 1 leader_uuid: "447c594df2cc412789f5ab82d92a0963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } } }
I20251025 08:01:50.767858 29832 tablet_replica.cc:445] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:50.769227 29837 tablet_replica.cc:442] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b: TxnStatusTablet state changed. Reason: New leader 447c594df2cc412789f5ab82d92a0963. Latest consensus state: current_term: 1 leader_uuid: "447c594df2cc412789f5ab82d92a0963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } } }
I20251025 08:01:50.769332 29837 tablet_replica.cc:445] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:50.773161 29868 tablet_replica.cc:442] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "447c594df2cc412789f5ab82d92a0963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } } }
I20251025 08:01:50.773298 29868 tablet_replica.cc:445] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: This TxnStatusTablet replica's current role is: LEADER
I20251025 08:01:50.773411 29868 tablet_replica.cc:442] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "447c594df2cc412789f5ab82d92a0963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } } }
I20251025 08:01:50.773480 29868 tablet_replica.cc:445] T cd34e9d70acb4c57bc206a65767e583d P 447c594df2cc412789f5ab82d92a0963: This TxnStatusTablet replica's current role is: LEADER
I20251025 08:01:50.775523 29832 tablet_replica.cc:442] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "447c594df2cc412789f5ab82d92a0963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } } }
I20251025 08:01:50.775637 29832 tablet_replica.cc:445] T cd34e9d70acb4c57bc206a65767e583d P 10130032710a4718b7d2ced43921f4a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:50.778108 29837 tablet_replica.cc:442] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "447c594df2cc412789f5ab82d92a0963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 } } peers { permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 } } peers { permanent_uuid: "c4c28a25ba7e4cb0b07c37f7c45a272b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 34567 } } }
I20251025 08:01:50.778219 29837 tablet_replica.cc:445] T cd34e9d70acb4c57bc206a65767e583d P c4c28a25ba7e4cb0b07c37f7c45a272b: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:50.778939 29840 mvcc.cc:204] Tried to move back new op lower bound from 7214609656888197120 to 7214609656824061952. Current Snapshot: MvccSnapshot[applied={T|T < 7214609656888197120 or (T in {7214609656888197120})}]
I20251025 08:01:50.788370 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.785517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:37087 (local address 127.27.132.3:36613)
1025 08:01:50.785733 (+ 216us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.785749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.785850 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.786113 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.786116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.786123 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:01:50.786338 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.786344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.787103 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.787105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.787591 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.787599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.787717 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.787766 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:01:50.788106 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:01:50.788113 (+ 7us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.788175 (+ 62us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.788251 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20251025 08:01:50.788370 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.785591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:36613 (local address 127.27.132.2:37087)
1025 08:01:50.785714 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.785718 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.785809 (+ 91us) server_negotiation.cc:408] Connection header received
1025 08:01:50.785967 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.785970 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.786014 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.786097 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:50.786451 (+ 354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.786974 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.787716 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.787906 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.787947 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:50.788018 (+ 71us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:50.788086 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.788191 (+ 105us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.788233 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 08:01:50.796283 29613 raft_consensus.cc:1275] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963 [term 1 FOLLOWER]: Refusing update from remote peer c4c28a25ba7e4cb0b07c37f7c45a272b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:50.796578 29471 raft_consensus.cc:1275] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1 [term 1 FOLLOWER]: Refusing update from remote peer c4c28a25ba7e4cb0b07c37f7c45a272b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:50.796725 29845 consensus_queue.cc:1048] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [LEADER]: Connected to new peer: Peer: permanent_uuid: "447c594df2cc412789f5ab82d92a0963" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 37087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:50.796979 29837 consensus_queue.cc:1048] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10130032710a4718b7d2ced43921f4a1" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:50.809005 29841 mvcc.cc:204] Tried to move back new op lower bound from 7214609657017593856 to 7214609656368099328. Current Snapshot: MvccSnapshot[applied={T|T < 7214609657017593856}]
I20251025 08:01:50.811633 29842 mvcc.cc:204] Tried to move back new op lower bound from 7214609657017593856 to 7214609656368099328. Current Snapshot: MvccSnapshot[applied={T|T < 7214609657017593856}]
I20251025 08:01:50.829594 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.825432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:60357 (local address 127.27.132.3:34567)
1025 08:01:50.825570 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.825574 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.825856 (+ 282us) server_negotiation.cc:408] Connection header received
1025 08:01:50.825982 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.825985 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.826113 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.826213 (+ 100us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:50.826872 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.827958 (+ 1086us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.828688 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.828880 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.828939 (+ 59us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:50.829008 (+ 69us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:50.829075 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.829373 (+ 298us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.829452 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251025 08:01:50.830405 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.825365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:34567 (local address 127.27.132.2:60357)
1025 08:01:50.825770 (+ 405us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:50.825783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:50.825894 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:50.826353 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:50.826357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:50.826367 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:01:50.826689 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.826699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.828086 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.828088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:50.828564 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:50.828571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.828696 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.828749 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:01:50.829163 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:01:50.829178 (+ 15us) client_negotiation.cc:769] Sending connection context
1025 08:01:50.829951 (+ 773us) client_negotiation.cc:240] Negotiation successful
1025 08:01:50.830033 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":149,"threads_started":1}
I20251025 08:01:50.855804 29312 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35900:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:50.856003 29312 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35900:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:50.858942 29312 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 1d85f35b610340ef8e65d27c2a6f6db5: Sending DeleteTablet for 3 replicas of tablet 6e84b15ba9b84ade9bc7caadf68e936a
I20251025 08:01:50.859625 29446 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e84b15ba9b84ade9bc7caadf68e936a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:50 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:49394
I20251025 08:01:50.859819 29852 tablet_replica.cc:333] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1: stopping tablet replica
I20251025 08:01:50.859925 29852 raft_consensus.cc:2243] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:01:50.859922 29593 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e84b15ba9b84ade9bc7caadf68e936a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:50 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:39566
I20251025 08:01:50.860023 29852 raft_consensus.cc:2272] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:50.860124 29850 tablet_replica.cc:333] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963: stopping tablet replica
I20251025 08:01:50.860169 29735 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e84b15ba9b84ade9bc7caadf68e936a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:50 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:59554
I20251025 08:01:50.860250 29850 raft_consensus.cc:2243] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:01:50.860355 29851 tablet_replica.cc:333] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b: stopping tablet replica
I20251025 08:01:50.860356 29850 raft_consensus.cc:2272] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:50.860466 29851 raft_consensus.cc:2243] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:01:50.860656 29851 raft_consensus.cc:2272] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:50.860843 29852 ts_tablet_manager.cc:1916] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:50.861519 29851 ts_tablet_manager.cc:1916] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:50.861967 29850 ts_tablet_manager.cc:1916] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:50.863569 29852 ts_tablet_manager.cc:1929] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:01:50.863663 29852 log.cc:1199] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-0/wal/wals/6e84b15ba9b84ade9bc7caadf68e936a
I20251025 08:01:50.863898 29851 ts_tablet_manager.cc:1929] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:01:50.863936 29852 ts_tablet_manager.cc:1950] T 6e84b15ba9b84ade9bc7caadf68e936a P 10130032710a4718b7d2ced43921f4a1: Deleting consensus metadata
I20251025 08:01:50.864001 29851 log.cc:1199] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/6e84b15ba9b84ade9bc7caadf68e936a
I20251025 08:01:50.864262 29851 ts_tablet_manager.cc:1950] T 6e84b15ba9b84ade9bc7caadf68e936a P c4c28a25ba7e4cb0b07c37f7c45a272b: Deleting consensus metadata
I20251025 08:01:50.864683 29850 ts_tablet_manager.cc:1929] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:01:50.864691 29299 catalog_manager.cc:4985] TS 10130032710a4718b7d2ced43921f4a1 (127.27.132.1:34023): tablet 6e84b15ba9b84ade9bc7caadf68e936a (table test-table [id=4b761ac37e454a55aff589850ce3e3aa]) successfully deleted
I20251025 08:01:50.864766 29850 log.cc:1199] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/6e84b15ba9b84ade9bc7caadf68e936a
I20251025 08:01:50.864933 29299 catalog_manager.cc:4985] TS c4c28a25ba7e4cb0b07c37f7c45a272b (127.27.132.3:34567): tablet 6e84b15ba9b84ade9bc7caadf68e936a (table test-table [id=4b761ac37e454a55aff589850ce3e3aa]) successfully deleted
I20251025 08:01:50.865031 29850 ts_tablet_manager.cc:1950] T 6e84b15ba9b84ade9bc7caadf68e936a P 447c594df2cc412789f5ab82d92a0963: Deleting consensus metadata
I20251025 08:01:50.865701 29296 catalog_manager.cc:4985] TS 447c594df2cc412789f5ab82d92a0963 (127.27.132.2:37087): tablet 6e84b15ba9b84ade9bc7caadf68e936a (table test-table [id=4b761ac37e454a55aff589850ce3e3aa]) successfully deleted
Oct 25 08:01:50 dist-test-slave-lq8l krb5kdc[29264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.132.1@KRBTEST.COM
I20251025 08:01:50.875890 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.861856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.27.132.1:34023)
1025 08:01:50.863526 (+ 1670us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.863531 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.864039 (+ 508us) server_negotiation.cc:408] Connection header received
1025 08:01:50.864437 (+ 398us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.864441 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.864514 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.864570 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:50.866006 (+ 1436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.866913 (+ 907us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.868324 (+ 1411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.868655 (+ 331us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.872147 (+ 3492us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:50.872174 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:50.872187 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:50.872217 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:50.874018 (+ 1801us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.874461 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.874470 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.874474 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.874558 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.874843 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.874846 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.874848 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.875052 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:50.875272 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.875492 (+ 220us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.875674 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1562}
W20251025 08:01:50.876451 29495 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49406
I20251025 08:01:50.885399 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:50.878051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.27.132.62:40031)
1025 08:01:50.878195 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:50.878200 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:50.878609 (+ 409us) server_negotiation.cc:408] Connection header received
1025 08:01:50.878721 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:50.878725 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:50.878781 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:50.878852 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:50.879689 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.880189 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:50.880806 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:50.881025 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:50.882211 (+ 1186us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:50.882231 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:50.882234 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:50.882382 (+ 148us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:50.883909 (+ 1527us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.884306 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.884309 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.884311 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.884354 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:50.884572 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:50.884575 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:50.884577 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:50.884736 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:50.884865 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:50.885108 (+ 243us) server_negotiation.cc:300] Negotiation successful
1025 08:01:50.885223 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20251025 08:01:50.885849 29312 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35926
I20251025 08:01:50.887058 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 29386
I20251025 08:01:50.897853 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 29528
I20251025 08:01:50.904994 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 29670
I20251025 08:01:50.912478 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 29280
2025-10-25T08:01:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3649 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:01:50 dist-test-slave-lq8l krb5kdc[29890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:01:50 dist-test-slave-lq8l krb5kdc[29890](info): set up 2 sockets
Oct 25 08:01:50 dist-test-slave-lq8l krb5kdc[29890](info): commencing operation
krb5kdc: starting...
W20251025 08:01:52.908160 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.972s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:01:52 dist-test-slave-lq8l krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:01:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:01:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:53.043033 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:46883
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:53.153059 29906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:53.153301 29906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:53.153393 29906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:53.156630 29906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:01:53.156697 29906 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251025 08:01:53.156724 29906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:53.156761 29906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:53.156800 29906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:01:53.156839 29906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:01:53.161190 29906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:46883
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:53.162295 29906 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:53.163157 29906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:53.168780 29914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:53.168783 29911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:53.169077 29906 server_base.cc:1047] running on GCE node
W20251025 08:01:53.168783 29912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:53.169596 29906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:53.170503 29906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:53.171659 29906 hybrid_clock.cc:648] HybridClock initialized: now 1761379313171649 us; error 25 us; skew 500 ppm
Oct 25 08:01:53 dist-test-slave-lq8l krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:53.174407 29906 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:01:53.175514 29906 webserver.cc:492] Webserver started at http://127.27.132.62:43063/ using document root <none> and password file <none>
I20251025 08:01:53.176069 29906 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:53.176126 29906 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:53.176307 29906 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:53.177994 29906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "fb647e644971498c88b4a76a0d6a4069"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "ea817257ba9039a2b658c2a98f6f787c"
server_key_iv: "2b5601222cc83d7da6b0ffac22eaf00f"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.178471 29906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "fb647e644971498c88b4a76a0d6a4069"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "ea817257ba9039a2b658c2a98f6f787c"
server_key_iv: "2b5601222cc83d7da6b0ffac22eaf00f"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.181843 29906 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:01:53.184031 29921 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:53.185005 29906 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:01:53.185117 29906 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "fb647e644971498c88b4a76a0d6a4069"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "ea817257ba9039a2b658c2a98f6f787c"
server_key_iv: "2b5601222cc83d7da6b0ffac22eaf00f"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.185204 29906 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:53.203078 29906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:53.209115 29906 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:53.209311 29906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:53.217185 29906 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:46883
I20251025 08:01:53.217181 29983 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:46883 every 8 connection(s)
I20251025 08:01:53.218214 29906 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:01:53.218937 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 29906
I20251025 08:01:53.219060 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:01:53.219300 28176 external_mini_cluster.cc:1468] Setting key c0ab587d90ba13889c72e883a5455256
I20251025 08:01:53.221539 29984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:53.226783 29984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069: Bootstrap starting.
Oct 25 08:01:53 dist-test-slave-lq8l krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:53.229154 29984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:53.229843 29984 log.cc:826] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:53.231827 29984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069: No bootstrap required, opened a new log
I20251025 08:01:53.232208 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.220636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.27.132.62:46883)
1025 08:01:53.221049 (+ 413us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:53.221059 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:53.221092 (+ 33us) server_negotiation.cc:408] Connection header received
1025 08:01:53.222169 (+ 1077us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:53.222189 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:53.222473 (+ 284us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:53.222804 (+ 331us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:53.223752 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.224500 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.225158 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.225439 (+ 281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.227787 (+ 2348us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:53.227826 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:53.227836 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:53.227868 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:53.229689 (+ 1821us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.230080 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.230085 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.230090 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.230163 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.230433 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.230437 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.230439 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.230815 (+ 376us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:53.230959 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:53.231238 (+ 279us) server_negotiation.cc:300] Negotiation successful
1025 08:01:53.231604 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":141,"threads_started":1}
I20251025 08:01:53.234910 29984 raft_consensus.cc:359] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb647e644971498c88b4a76a0d6a4069" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46883 } }
I20251025 08:01:53.235106 29984 raft_consensus.cc:385] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:53.235174 29984 raft_consensus.cc:740] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb647e644971498c88b4a76a0d6a4069, State: Initialized, Role: FOLLOWER
I20251025 08:01:53.235606 29984 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb647e644971498c88b4a76a0d6a4069" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46883 } }
I20251025 08:01:53.235734 29984 raft_consensus.cc:399] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:01:53.235800 29984 raft_consensus.cc:493] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:01:53.235894 29984 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:53.236702 29984 raft_consensus.cc:515] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb647e644971498c88b4a76a0d6a4069" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46883 } }
I20251025 08:01:53.237025 29984 leader_election.cc:304] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb647e644971498c88b4a76a0d6a4069; no voters:
I20251025 08:01:53.237303 29984 leader_election.cc:290] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:01:53.237418 29989 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:53.237615 29989 raft_consensus.cc:697] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [term 1 LEADER]: Becoming Leader. State: Replica: fb647e644971498c88b4a76a0d6a4069, State: Running, Role: LEADER
I20251025 08:01:53.237887 29989 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb647e644971498c88b4a76a0d6a4069" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46883 } }
I20251025 08:01:53.238371 29984 sys_catalog.cc:565] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:01:53.239400 29991 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb647e644971498c88b4a76a0d6a4069. Latest consensus state: current_term: 1 leader_uuid: "fb647e644971498c88b4a76a0d6a4069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb647e644971498c88b4a76a0d6a4069" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46883 } } }
I20251025 08:01:53.239516 29991 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [sys.catalog]: This master's current role is: LEADER
I20251025 08:01:53.239809 29990 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb647e644971498c88b4a76a0d6a4069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb647e644971498c88b4a76a0d6a4069" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46883 } } }
I20251025 08:01:53.239905 29990 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069 [sys.catalog]: This master's current role is: LEADER
W20251025 08:01:53.242995 30005 catalog_manager.cc:1568] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 08:01:53.243060 30005 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 08:01:53.243134 29998 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:01:53.243870 29998 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:01:53.248467 29998 catalog_manager.cc:1357] Generated new cluster ID: 1b07fd5390124662b56fd7bf9e73cc00
I20251025 08:01:53.248524 29998 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:01:53.256429 29998 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:01:53.257632 29998 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:01:53.264067 29998 catalog_manager.cc:6022] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069: Generated new TSK 0
I20251025 08:01:53.264986 29998 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:53.320256 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:46883
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:53.428026 30012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:53.428236 30012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:53.428299 30012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:53.431507 30012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:53.431567 30012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:53.431620 30012 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251025 08:01:53.431646 30012 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 08:01:53.431691 30012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:01:53.435871 30012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.132.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:53.436931 30012 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:53.437755 30012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:53.444260 30018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:53.444252 30017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:53.444247 30020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:53.444473 30012 server_base.cc:1047] running on GCE node
I20251025 08:01:53.444926 30012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:53.445465 30012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:53.446609 30012 hybrid_clock.cc:648] HybridClock initialized: now 1761379313446601 us; error 26 us; skew 500 ppm
Oct 25 08:01:53 dist-test-slave-lq8l krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:53.449357 30012 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:01:53.450420 30012 webserver.cc:492] Webserver started at http://127.27.132.1:37841/ using document root <none> and password file <none>
I20251025 08:01:53.451035 30012 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:53.451086 30012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:53.451244 30012 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:53.452947 30012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "cfb073222c6c4801915af8ec1960f147"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "c55a5b6d40c1c4115b214ddf1c3ac61d"
server_key_iv: "b298999310b7fcfcc9bbf25042fa5d7e"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.453375 30012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "cfb073222c6c4801915af8ec1960f147"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "c55a5b6d40c1c4115b214ddf1c3ac61d"
server_key_iv: "b298999310b7fcfcc9bbf25042fa5d7e"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.456736 30012 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:01:53.458920 30027 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:53.460012 30012 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:01:53.460106 30012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "cfb073222c6c4801915af8ec1960f147"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "c55a5b6d40c1c4115b214ddf1c3ac61d"
server_key_iv: "b298999310b7fcfcc9bbf25042fa5d7e"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.460207 30012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:53.480173 30012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:53.483232 30012 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:53.483429 30012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:53.484918 30012 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:53.484969 30012 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:53.485174 30012 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:53.485224 30012 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 08:01:53 dist-test-slave-lq8l krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:53.496181 30012 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:42569
I20251025 08:01:53.496208 30142 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:42569 every 8 connection(s)
I20251025 08:01:53.497345 30012 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:01:53.498405 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.485664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:43875 (local address 127.27.132.62:46883)
1025 08:01:53.485806 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:53.485810 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:53.486664 (+ 854us) server_negotiation.cc:408] Connection header received
1025 08:01:53.487587 (+ 923us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:53.487591 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:53.487665 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:53.487743 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:53.489457 (+ 1714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.490249 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.491309 (+ 1060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.491545 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.494502 (+ 2957us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:53.494533 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:53.494541 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:53.494582 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:53.496441 (+ 1859us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.496897 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.496901 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.496903 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.496957 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.497202 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.497204 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.497210 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.497446 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:53.497548 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:53.498112 (+ 564us) server_negotiation.cc:300] Negotiation successful
1025 08:01:53.498247 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251025 08:01:53.499047 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.486016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46883 (local address 127.27.132.1:43875)
1025 08:01:53.486520 (+ 504us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:53.486556 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:53.487328 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:53.487913 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:53.487921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:53.488505 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:53.489284 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.489305 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.490458 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.490465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:53.491164 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.491174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.491477 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.492207 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:53.492240 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:53.494284 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:53.496556 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.496560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.496569 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.496798 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.497046 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.497049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.497050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.497118 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.497562 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:53.497567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:53.497842 (+ 275us) client_negotiation.cc:769] Sending connection context
1025 08:01:53.498071 (+ 229us) client_negotiation.cc:240] Negotiation successful
1025 08:01:53.498333 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":145,"threads_started":1}
I20251025 08:01:53.500257 30143 heartbeater.cc:344] Connected to a master server at 127.27.132.62:46883
I20251025 08:01:53.500442 30143 heartbeater.cc:461] Registering TS with master...
I20251025 08:01:53.501042 30143 heartbeater.cc:507] Master 127.27.132.62:46883 requested a full tablet report, sending...
I20251025 08:01:53.502681 29937 ts_manager.cc:194] Registered new tserver with Master: cfb073222c6c4801915af8ec1960f147 (127.27.132.1:42569)
I20251025 08:01:53.504189 29937 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:43875
I20251025 08:01:53.506215 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 30012
I20251025 08:01:53.506341 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:01:53.506583 28176 external_mini_cluster.cc:1468] Setting key ef7071476aebee3b710b67f53610ec37
I20251025 08:01:53.510453 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.502487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46883 (local address 127.27.132.1:57339)
1025 08:01:53.502781 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:53.502797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:53.502905 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:53.503161 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:53.503164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:53.503400 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:53.503697 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.503707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.504480 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.504484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:53.505042 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.505051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.505169 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.505729 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:53.505742 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:53.506084 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:53.508986 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.508993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.508999 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.509250 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.509488 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.509494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.509499 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.509564 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.510018 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:53.510020 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:53.510097 (+ 77us) client_negotiation.cc:769] Sending connection context
1025 08:01:53.510180 (+ 83us) client_negotiation.cc:240] Negotiation successful
1025 08:01:53.510289 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20251025 08:01:53.510677 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.502601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:57339 (local address 127.27.132.62:46883)
1025 08:01:53.502743 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:53.502747 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:53.502868 (+ 121us) server_negotiation.cc:408] Connection header received
1025 08:01:53.503001 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:53.503003 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:53.503055 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:53.503125 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:53.503851 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.504357 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.505179 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.505368 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.507199 (+ 1831us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:53.507215 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:53.507217 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:53.507240 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:53.508865 (+ 1625us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.509350 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.509353 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.509354 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.509403 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.509694 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.509701 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.509704 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.509918 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:53.510035 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:53.510259 (+ 224us) server_negotiation.cc:300] Negotiation successful
1025 08:01:53.510435 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:53.552130 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:46883
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:53.660566 30154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:53.660806 30154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:53.660876 30154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:53.664155 30154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:53.664225 30154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:53.664281 30154 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251025 08:01:53.664300 30154 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 08:01:53.664348 30154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:01:53.668809 30154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.132.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:53.669889 30154 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:53.670732 30154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:53.677508 30159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:53.677635 30160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:53.677901 30162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:53.678448 30154 server_base.cc:1047] running on GCE node
I20251025 08:01:53.678964 30154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:53.679744 30154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:53.680893 30154 hybrid_clock.cc:648] HybridClock initialized: now 1761379313680864 us; error 37 us; skew 500 ppm
Oct 25 08:01:53 dist-test-slave-lq8l krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:53.684577 30154 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:01:53.685973 30154 webserver.cc:492] Webserver started at http://127.27.132.2:42427/ using document root <none> and password file <none>
I20251025 08:01:53.686673 30154 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:53.686743 30154 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:53.686959 30154 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:53.689677 30154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "9781542c5eaf4f68be13b04698643193"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "acca19864040dbe496d56f06dde9a1fa"
server_key_iv: "6e107539c69991b2f379f26e398c06eb"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.690320 30154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "9781542c5eaf4f68be13b04698643193"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "acca19864040dbe496d56f06dde9a1fa"
server_key_iv: "6e107539c69991b2f379f26e398c06eb"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.694808 30154 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20251025 08:01:53.698098 30169 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:53.699448 30154 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20251025 08:01:53.699576 30154 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "9781542c5eaf4f68be13b04698643193"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "acca19864040dbe496d56f06dde9a1fa"
server_key_iv: "6e107539c69991b2f379f26e398c06eb"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.699693 30154 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:53.721256 30154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:53.724327 30154 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:53.724515 30154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:53.725973 30154 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:53.726028 30154 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:53.726091 30154 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:53.726121 30154 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 08:01:53 dist-test-slave-lq8l krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:53.736716 30154 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:41373
I20251025 08:01:53.736840 30284 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:41373 every 8 connection(s)
I20251025 08:01:53.737996 30154 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:01:53.738139 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 30154
I20251025 08:01:53.738236 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:01:53.738521 28176 external_mini_cluster.cc:1468] Setting key 86e033ac6a6af1cebcff452cf7c38bd0
I20251025 08:01:53.740484 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.726685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:50121 (local address 127.27.132.62:46883)
1025 08:01:53.726845 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:53.726851 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:53.727484 (+ 633us) server_negotiation.cc:408] Connection header received
1025 08:01:53.728473 (+ 989us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:53.728485 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:53.728554 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:53.728668 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:53.730297 (+ 1629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.731150 (+ 853us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.732242 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.732432 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.736182 (+ 3750us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:53.736201 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:53.736203 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:53.736230 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:53.738141 (+ 1911us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.738788 (+ 647us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.738791 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.738793 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.738850 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.739188 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.739191 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.739193 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.739431 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:53.739545 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:53.740150 (+ 605us) server_negotiation.cc:300] Negotiation successful
1025 08:01:53.740284 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251025 08:01:53.740947 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.726909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46883 (local address 127.27.132.2:50121)
1025 08:01:53.727349 (+ 440us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:53.727390 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:53.728213 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:53.728828 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:53.728838 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:53.729395 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:53.730111 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.730131 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.731277 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.731281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:53.732089 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.732099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.732617 (+ 518us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.733307 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:53.733346 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:53.736028 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:53.738300 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.738306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.738317 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.738643 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.738964 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.738967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.738969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.739101 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.739555 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:53.739567 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:53.739862 (+ 295us) client_negotiation.cc:769] Sending connection context
1025 08:01:53.740089 (+ 227us) client_negotiation.cc:240] Negotiation successful
1025 08:01:53.740317 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20251025 08:01:53.742174 30285 heartbeater.cc:344] Connected to a master server at 127.27.132.62:46883
I20251025 08:01:53.742326 30285 heartbeater.cc:461] Registering TS with master...
I20251025 08:01:53.742802 30285 heartbeater.cc:507] Master 127.27.132.62:46883 requested a full tablet report, sending...
I20251025 08:01:53.743734 29938 ts_manager.cc:194] Registered new tserver with Master: 9781542c5eaf4f68be13b04698643193 (127.27.132.2:41373)
I20251025 08:01:53.744441 29938 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:50121
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
I20251025 08:01:53.750895 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.744337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:39719 (local address 127.27.132.62:46883)
1025 08:01:53.744489 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:53.744495 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:53.744768 (+ 273us) server_negotiation.cc:408] Connection header received
1025 08:01:53.744896 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:53.744899 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:53.744944 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:53.745028 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:53.745635 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.746104 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.746685 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.746885 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.747843 (+ 958us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:53.747857 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:53.747860 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:53.747880 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:53.749406 (+ 1526us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.749847 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.749851 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.749853 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.749897 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.750112 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.750115 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.750117 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.750288 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:53.750392 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:53.750632 (+ 240us) server_negotiation.cc:300] Negotiation successful
1025 08:01:53.750744 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 08:01:53.750949 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.744257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46883 (local address 127.27.132.2:39719)
1025 08:01:53.744673 (+ 416us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:53.744690 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:53.744799 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:53.745057 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:53.745061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:53.745277 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:53.745522 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.745529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.746209 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.746212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:53.746581 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.746588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.746700 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.747316 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:53.747437 (+ 121us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:53.747727 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:53.749553 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.749562 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.749564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.749749 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.749984 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.749987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.749989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.750032 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.750391 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:53.750396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:53.750479 (+ 83us) client_negotiation.cc:769] Sending connection context
1025 08:01:53.750573 (+ 94us) client_negotiation.cc:240] Negotiation successful
1025 08:01:53.750727 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:01:53.786151 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:46883
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:01:53.894205 30296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:01:53.894464 30296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:01:53.894562 30296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:01:53.897892 30296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:01:53.897974 30296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:01:53.898064 30296 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251025 08:01:53.898094 30296 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 08:01:53.898157 30296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:01:53.902541 30296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.132.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:01:53.903717 30296 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:01:53.904584 30296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:01:53.911201 30302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:01:53.911319 30301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:53.911478 30296 server_base.cc:1047] running on GCE node
W20251025 08:01:53.911612 30304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:01:53.912006 30296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:01:53.912549 30296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:01:53.913718 30296 hybrid_clock.cc:648] HybridClock initialized: now 1761379313913700 us; error 37 us; skew 500 ppm
Oct 25 08:01:53 dist-test-slave-lq8l krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:01:53.916564 30296 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:01:53.917697 30296 webserver.cc:492] Webserver started at http://127.27.132.3:33265/ using document root <none> and password file <none>
I20251025 08:01:53.918246 30296 fs_manager.cc:362] Metadata directory not provided
I20251025 08:01:53.918303 30296 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:01:53.918503 30296 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:01:53.920223 30296 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "cf1b7a58056d4a0083000de1210e4d8f"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "31c40e35111e3488e1583bf6b222559f"
server_key_iv: "ead318a0e8294c5b395474cb395039f7"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.920696 30296 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "cf1b7a58056d4a0083000de1210e4d8f"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "31c40e35111e3488e1583bf6b222559f"
server_key_iv: "ead318a0e8294c5b395474cb395039f7"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.924144 30296 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251025 08:01:53.926347 30311 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:53.927301 30296 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:01:53.927457 30296 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "cf1b7a58056d4a0083000de1210e4d8f"
format_stamp: "Formatted at 2025-10-25 08:01:53 on dist-test-slave-lq8l"
server_key: "31c40e35111e3488e1583bf6b222559f"
server_key_iv: "ead318a0e8294c5b395474cb395039f7"
server_key_version: "encryptionkey@0"
I20251025 08:01:53.927548 30296 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:01:53.953694 30296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:01:53.956571 30296 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:01:53.956763 30296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:01:53.958237 30296 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:01:53.958292 30296 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:01:53.958355 30296 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:01:53.958384 30296 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 08:01:53 dist-test-slave-lq8l krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:53.970161 30296 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:40623
I20251025 08:01:53.970202 30426 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:40623 every 8 connection(s)
I20251025 08:01:53.971123 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.958924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:43841 (local address 127.27.132.62:46883)
1025 08:01:53.959083 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:53.959089 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:53.959932 (+ 843us) server_negotiation.cc:408] Connection header received
1025 08:01:53.961047 (+ 1115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:53.961051 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:53.961104 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:53.961178 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:53.963302 (+ 2124us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.963853 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.964683 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.964935 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.967227 (+ 2292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:53.967252 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:53.967255 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:53.967292 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:53.969102 (+ 1810us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.969669 (+ 567us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.969672 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.969674 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.969725 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.970055 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.970059 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.970061 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.970248 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:53.970346 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:53.970908 (+ 562us) server_negotiation.cc:300] Negotiation successful
1025 08:01:53.971006 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251025 08:01:53.971359 30296 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:01:53.971915 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 30296
I20251025 08:01:53.972021 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:01:53.971961 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.959182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46883 (local address 127.27.132.3:43841)
1025 08:01:53.959753 (+ 571us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:53.959809 (+ 56us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:53.960809 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:53.961371 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:53.961385 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:53.962325 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:53.963120 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.963141 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.963982 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.963985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:53.964530 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.964540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.964813 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.965456 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:53.965489 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:53.967075 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:53.969208 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.969213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.969224 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.969568 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.969818 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.969821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.969823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.969944 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.970379 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:53.970385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:53.970643 (+ 258us) client_negotiation.cc:769] Sending connection context
1025 08:01:53.970870 (+ 227us) client_negotiation.cc:240] Negotiation successful
1025 08:01:53.971151 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":168,"threads_started":1}
I20251025 08:01:53.972282 28176 external_mini_cluster.cc:1468] Setting key 1bee241f3b341ea2cb7211dc98087fb5
I20251025 08:01:53.973294 30427 heartbeater.cc:344] Connected to a master server at 127.27.132.62:46883
I20251025 08:01:53.973464 30427 heartbeater.cc:461] Registering TS with master...
I20251025 08:01:53.973959 30427 heartbeater.cc:507] Master 127.27.132.62:46883 requested a full tablet report, sending...
I20251025 08:01:53.974942 29937 ts_manager.cc:194] Registered new tserver with Master: cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623)
I20251025 08:01:53.975708 29937 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:43841
I20251025 08:01:53.981063 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.975371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46883 (local address 127.27.132.3:52599)
1025 08:01:53.975587 (+ 216us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:53.975599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:53.975687 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:53.975935 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:53.975939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:53.976135 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:53.976338 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.976343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.977260 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.977263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:53.977627 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:53.977635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.977722 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.978234 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:53.978246 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:53.978525 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:53.979976 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.979979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.979981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.980136 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.980320 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:53.980322 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:53.980324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:53.980361 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:53.980707 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:53.980709 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:53.980770 (+ 61us) client_negotiation.cc:769] Sending connection context
1025 08:01:53.980842 (+ 72us) client_negotiation.cc:240] Negotiation successful
1025 08:01:53.980932 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20251025 08:01:53.981076 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:53.975311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:52599 (local address 127.27.132.62:46883)
1025 08:01:53.975443 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:53.975446 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:53.975678 (+ 232us) server_negotiation.cc:408] Connection header received
1025 08:01:53.975771 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:53.975775 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:53.975816 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:53.975889 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:53.976481 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.977160 (+ 679us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:53.977729 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:53.977855 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:53.978617 (+ 762us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:53.978629 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:53.978631 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:53.978647 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:53.979879 (+ 1232us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.980219 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.980222 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.980223 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.980260 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:53.980469 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:53.980472 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:53.980474 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:53.980620 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:53.980715 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:53.980885 (+ 170us) server_negotiation.cc:300] Negotiation successful
1025 08:01:53.980973 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251025 08:01:53.986668 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 25 08:01:53 dist-test-slave-lq8l krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 25 08:01:54 dist-test-slave-lq8l krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:54.010428 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.002297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.27.132.62:46883)
1025 08:01:54.002461 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.002466 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.002478 (+ 12us) server_negotiation.cc:408] Connection header received
1025 08:01:54.002519 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.002521 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.002571 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.002638 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:54.003597 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.004112 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.004826 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.005020 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.007251 (+ 2231us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:54.007270 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:54.007272 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:54.007307 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:54.008799 (+ 1492us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:54.009181 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:54.009184 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:54.009185 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:54.009230 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:54.009537 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:54.009541 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:54.009542 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:54.009720 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:54.009847 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.010083 (+ 236us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.010191 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251025 08:01:54.013262 29937 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:01:54.015755 29937 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:01:54.028903 30444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.024225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:42569 (local address 127.0.0.1:36124)
1025 08:01:54.024613 (+ 388us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:54.024658 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:54.024821 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:54.025648 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:54.025655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:54.025679 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:54.026003 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.026012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.027381 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.027385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:54.028372 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.028380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.028490 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.028553 (+ 63us) client_negotiation.cc:769] Sending connection context
1025 08:01:54.028657 (+ 104us) client_negotiation.cc:240] Negotiation successful
1025 08:01:54.028754 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
I20251025 08:01:54.029093 30448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.024814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:41373 (local address 127.0.0.1:53380)
1025 08:01:54.025411 (+ 597us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:54.025426 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:54.025521 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:54.026238 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:54.026241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:54.026260 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:54.026523 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.026532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.027815 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.027819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:54.028705 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.028715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.028826 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.028844 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 08:01:54.028894 (+ 50us) client_negotiation.cc:240] Negotiation successful
1025 08:01:54.028952 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":127,"threads_started":1}
I20251025 08:01:54.029343 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.025190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:40623 (local address 127.0.0.1:55724)
1025 08:01:54.025808 (+ 618us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:54.025822 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:54.025909 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:54.026601 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:54.026606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:54.026626 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:54.026811 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.026816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.028088 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.028091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:54.028974 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.028991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.029102 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.029116 (+ 14us) client_negotiation.cc:769] Sending connection context
1025 08:01:54.029159 (+ 43us) client_negotiation.cc:240] Negotiation successful
1025 08:01:54.029217 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":90,"threads_started":1}
I20251025 08:01:54.029587 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.024381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.27.132.1:42569)
1025 08:01:54.024806 (+ 425us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.024816 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.024848 (+ 32us) server_negotiation.cc:408] Connection header received
1025 08:01:54.025059 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.025068 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.025288 (+ 220us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.025451 (+ 163us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:54.026158 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.027215 (+ 1057us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.028561 (+ 1346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.029118 (+ 557us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.029257 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.029324 (+ 67us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.029437 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":91,"threads_started":1}
I20251025 08:01:54.030337 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.025042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.27.132.2:41373)
1025 08:01:54.025796 (+ 754us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.025803 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.025828 (+ 25us) server_negotiation.cc:408] Connection header received
1025 08:01:54.025927 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.025936 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.026096 (+ 160us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.026256 (+ 160us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:54.026640 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.027649 (+ 1009us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.029205 (+ 1556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.030041 (+ 836us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.030123 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.030175 (+ 52us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.030233 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":128,"threads_started":1}
I20251025 08:01:54.030475 30449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.025602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.27.132.3:40623)
1025 08:01:54.026023 (+ 421us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.026029 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.026059 (+ 30us) server_negotiation.cc:408] Connection header received
1025 08:01:54.026143 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.026155 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.026309 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.026448 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:54.026933 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.027949 (+ 1016us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.029532 (+ 1583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.030064 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.030164 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.030225 (+ 61us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.030305 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":142,"threads_started":1}
I20251025 08:01:54.032980 30361 tablet_service.cc:1505] Processing CreateTablet for tablet 893dc67cfa0c4583a34115614d7f6924 (DEFAULT_TABLE table=test-table [id=e3bacd8db4014b69b4775b50d21fe148]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:54.032877 30077 tablet_service.cc:1505] Processing CreateTablet for tablet 893dc67cfa0c4583a34115614d7f6924 (DEFAULT_TABLE table=test-table [id=e3bacd8db4014b69b4775b50d21fe148]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:54.032881 30219 tablet_service.cc:1505] Processing CreateTablet for tablet 893dc67cfa0c4583a34115614d7f6924 (DEFAULT_TABLE table=test-table [id=e3bacd8db4014b69b4775b50d21fe148]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:54.033839 30361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 893dc67cfa0c4583a34115614d7f6924. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:54.033855 30219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 893dc67cfa0c4583a34115614d7f6924. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:54.033855 30077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 893dc67cfa0c4583a34115614d7f6924. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:54.039670 30450 tablet_bootstrap.cc:492] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f: Bootstrap starting.
I20251025 08:01:54.040988 30452 tablet_bootstrap.cc:492] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193: Bootstrap starting.
I20251025 08:01:54.040988 30451 tablet_bootstrap.cc:492] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147: Bootstrap starting.
I20251025 08:01:54.042923 30450 tablet_bootstrap.cc:654] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:54.043148 30451 tablet_bootstrap.cc:654] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:54.043148 30452 tablet_bootstrap.cc:654] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:54.044147 30450 log.cc:826] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:54.044155 30451 log.cc:826] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:54.044394 30452 log.cc:826] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193: Log is configured to *not* fsync() on all Append() calls
I20251025 08:01:54.045950 30450 tablet_bootstrap.cc:492] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f: No bootstrap required, opened a new log
I20251025 08:01:54.046057 30451 tablet_bootstrap.cc:492] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147: No bootstrap required, opened a new log
I20251025 08:01:54.046057 30452 tablet_bootstrap.cc:492] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193: No bootstrap required, opened a new log
I20251025 08:01:54.046263 30452 ts_tablet_manager.cc:1403] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251025 08:01:54.046263 30451 ts_tablet_manager.cc:1403] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251025 08:01:54.046336 30450 ts_tablet_manager.cc:1403] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20251025 08:01:54.049115 30450 raft_consensus.cc:359] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.049312 30450 raft_consensus.cc:385] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:54.049376 30450 raft_consensus.cc:740] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1b7a58056d4a0083000de1210e4d8f, State: Initialized, Role: FOLLOWER
I20251025 08:01:54.049531 30451 raft_consensus.cc:359] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.049531 30452 raft_consensus.cc:359] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.049741 30452 raft_consensus.cc:385] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:54.049741 30451 raft_consensus.cc:385] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:54.049800 30451 raft_consensus.cc:740] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb073222c6c4801915af8ec1960f147, State: Initialized, Role: FOLLOWER
I20251025 08:01:54.049800 30452 raft_consensus.cc:740] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9781542c5eaf4f68be13b04698643193, State: Initialized, Role: FOLLOWER
I20251025 08:01:54.049829 30450 consensus_queue.cc:260] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.050264 30452 consensus_queue.cc:260] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.050264 30451 consensus_queue.cc:260] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.050554 30427 heartbeater.cc:499] Master 127.27.132.62:46883 was elected leader, sending a full tablet report...
I20251025 08:01:54.050976 30143 heartbeater.cc:499] Master 127.27.132.62:46883 was elected leader, sending a full tablet report...
I20251025 08:01:54.051188 30452 ts_tablet_manager.cc:1434] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20251025 08:01:54.051541 30285 heartbeater.cc:499] Master 127.27.132.62:46883 was elected leader, sending a full tablet report...
I20251025 08:01:54.051795 30451 ts_tablet_manager.cc:1434] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20251025 08:01:54.053331 30450 ts_tablet_manager.cc:1434] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251025 08:01:54.081562 30457 raft_consensus.cc:493] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:01:54.081769 30457 raft_consensus.cc:515] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.082948 30457 leader_election.cc:290] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9781542c5eaf4f68be13b04698643193 (127.27.132.2:41373), cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623)
I20251025 08:01:54.087037 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.083181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:41373 (local address 127.27.132.1:51885)
1025 08:01:54.083537 (+ 356us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:54.083551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:54.083688 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:54.083977 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:54.083982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:54.084010 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:54.084862 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.084869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.085860 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.085864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:54.086603 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.086628 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.086737 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.086762 (+ 25us) client_negotiation.cc:769] Sending connection context
1025 08:01:54.086812 (+ 50us) client_negotiation.cc:240] Negotiation successful
1025 08:01:54.086870 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"spinlock_wait_cycles":33152,"thread_start_us":156,"threads_started":1}
I20251025 08:01:54.087070 30460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.083637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:40623 (local address 127.27.132.1:51625)
1025 08:01:54.083964 (+ 327us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:54.083977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:54.084084 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:54.084301 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:54.084305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:54.084327 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:54.084715 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.084725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.085847 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.085853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:54.086671 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.086680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.086799 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.086817 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 08:01:54.086867 (+ 50us) client_negotiation.cc:240] Negotiation successful
1025 08:01:54.086922 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"spinlock_wait_cycles":384,"thread_start_us":102,"threads_started":1}
I20251025 08:01:54.087559 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.083300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:51885 (local address 127.27.132.2:41373)
1025 08:01:54.083478 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.083485 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.083663 (+ 178us) server_negotiation.cc:408] Connection header received
1025 08:01:54.083810 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.083814 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.083875 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.083942 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:54.084997 (+ 1055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.085722 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.086749 (+ 1027us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.087247 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.087291 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.087381 (+ 90us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.087447 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251025 08:01:54.087623 30449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.083766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:51625 (local address 127.27.132.3:40623)
1025 08:01:54.083891 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.083895 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.084054 (+ 159us) server_negotiation.cc:408] Connection header received
1025 08:01:54.084171 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.084174 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.084214 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.084274 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:54.084869 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.085692 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.086798 (+ 1106us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.087312 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.087387 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.087446 (+ 59us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.087517 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251025 08:01:54.088138 30239 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "893dc67cfa0c4583a34115614d7f6924" candidate_uuid: "cfb073222c6c4801915af8ec1960f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9781542c5eaf4f68be13b04698643193" is_pre_election: true
I20251025 08:01:54.088140 30378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "893dc67cfa0c4583a34115614d7f6924" candidate_uuid: "cfb073222c6c4801915af8ec1960f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1b7a58056d4a0083000de1210e4d8f" is_pre_election: true
I20251025 08:01:54.088387 30239 raft_consensus.cc:2468] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfb073222c6c4801915af8ec1960f147 in term 0.
I20251025 08:01:54.088397 30378 raft_consensus.cc:2468] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfb073222c6c4801915af8ec1960f147 in term 0.
I20251025 08:01:54.088810 30028 leader_election.cc:304] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9781542c5eaf4f68be13b04698643193, cfb073222c6c4801915af8ec1960f147; no voters:
I20251025 08:01:54.089048 30457 raft_consensus.cc:2804] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:01:54.089143 30457 raft_consensus.cc:493] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:01:54.089195 30457 raft_consensus.cc:3060] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:54.090164 30457 raft_consensus.cc:515] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.090503 30457 leader_election.cc:290] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [CANDIDATE]: Term 1 election: Requested vote from peers 9781542c5eaf4f68be13b04698643193 (127.27.132.2:41373), cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623)
I20251025 08:01:54.090881 30239 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "893dc67cfa0c4583a34115614d7f6924" candidate_uuid: "cfb073222c6c4801915af8ec1960f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9781542c5eaf4f68be13b04698643193"
I20251025 08:01:54.090894 30378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "893dc67cfa0c4583a34115614d7f6924" candidate_uuid: "cfb073222c6c4801915af8ec1960f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1b7a58056d4a0083000de1210e4d8f"
I20251025 08:01:54.090993 30239 raft_consensus.cc:3060] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:54.091007 30378 raft_consensus.cc:3060] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:54.092020 30378 raft_consensus.cc:2468] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfb073222c6c4801915af8ec1960f147 in term 1.
I20251025 08:01:54.092046 30239 raft_consensus.cc:2468] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfb073222c6c4801915af8ec1960f147 in term 1.
I20251025 08:01:54.092300 30028 leader_election.cc:304] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9781542c5eaf4f68be13b04698643193, cfb073222c6c4801915af8ec1960f147; no voters:
I20251025 08:01:54.092481 30457 raft_consensus.cc:2804] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:54.092703 30457 raft_consensus.cc:697] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 1 LEADER]: Becoming Leader. State: Replica: cfb073222c6c4801915af8ec1960f147, State: Running, Role: LEADER
I20251025 08:01:54.093022 30457 consensus_queue.cc:237] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.096266 29938 catalog_manager.cc:5649] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfb073222c6c4801915af8ec1960f147 (127.27.132.1). New cstate: current_term: 1 leader_uuid: "cfb073222c6c4801915af8ec1960f147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } health_report { overall_health: UNKNOWN } } }
Oct 25 08:01:54 dist-test-slave-lq8l krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:54.132790 30468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.125124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46883 (local address 127.0.0.1:55724)
1025 08:01:54.125455 (+ 331us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:54.125470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:54.125596 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:54.125845 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:54.125848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:54.126302 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:01:54.126504 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.126511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.127207 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.127212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:54.127596 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.127604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.127713 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.128312 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:01:54.128326 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:01:54.129745 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:01:54.131269 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:54.131278 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:54.131283 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:54.131617 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:54.131876 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:01:54.131881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:01:54.131883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:01:54.131924 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:01:54.132238 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:01:54.132247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:01:54.132339 (+ 92us) client_negotiation.cc:769] Sending connection context
1025 08:01:54.132448 (+ 109us) client_negotiation.cc:240] Negotiation successful
1025 08:01:54.132598 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":129,"threads_started":1}
I20251025 08:01:54.132789 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.125243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.27.132.62:46883)
1025 08:01:54.125417 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.125422 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.125545 (+ 123us) server_negotiation.cc:408] Connection header received
1025 08:01:54.125707 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.125710 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.125749 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.125814 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:54.126645 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.127107 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.127706 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.127873 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.129870 (+ 1997us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:54.129884 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:54.129887 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:54.129908 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:54.131141 (+ 1233us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:54.131743 (+ 602us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:54.131747 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:54.131749 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:54.131798 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:54.132003 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:54.132006 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:54.132008 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:54.132141 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:54.132227 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.132498 (+ 271us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.132672 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":74}
I20251025 08:01:54.136657 29938 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:55724:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251025 08:01:54.137349 29938 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:01:54.142341 30077 tablet_service.cc:1505] Processing CreateTablet for tablet 26efcb1a9b634e6c8131ac3a70359f14 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e4263806ffa475d809055c347cd3072]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 08:01:54.142335 30219 tablet_service.cc:1505] Processing CreateTablet for tablet 26efcb1a9b634e6c8131ac3a70359f14 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e4263806ffa475d809055c347cd3072]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 08:01:54.142571 30361 tablet_service.cc:1505] Processing CreateTablet for tablet 26efcb1a9b634e6c8131ac3a70359f14 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e4263806ffa475d809055c347cd3072]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 08:01:54.142951 30077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26efcb1a9b634e6c8131ac3a70359f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:54.142993 30219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26efcb1a9b634e6c8131ac3a70359f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:54.143004 30361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26efcb1a9b634e6c8131ac3a70359f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:54.145202 30451 tablet_bootstrap.cc:492] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: Bootstrap starting.
I20251025 08:01:54.145668 30452 tablet_bootstrap.cc:492] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: Bootstrap starting.
I20251025 08:01:54.145699 30450 tablet_bootstrap.cc:492] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: Bootstrap starting.
I20251025 08:01:54.146306 30451 tablet_bootstrap.cc:654] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:54.146492 30450 tablet_bootstrap.cc:654] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:54.146711 30452 tablet_bootstrap.cc:654] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:54.147706 30451 tablet_bootstrap.cc:492] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: No bootstrap required, opened a new log
I20251025 08:01:54.147789 30451 ts_tablet_manager.cc:1403] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251025 08:01:54.147804 30450 tablet_bootstrap.cc:492] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: No bootstrap required, opened a new log
I20251025 08:01:54.147866 30450 ts_tablet_manager.cc:1403] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251025 08:01:54.148272 30452 tablet_bootstrap.cc:492] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: No bootstrap required, opened a new log
I20251025 08:01:54.148252 30451 raft_consensus.cc:359] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.148252 30450 raft_consensus.cc:359] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.148329 30452 ts_tablet_manager.cc:1403] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251025 08:01:54.148355 30451 raft_consensus.cc:385] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:54.148357 30450 raft_consensus.cc:385] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:54.148391 30451 raft_consensus.cc:740] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb073222c6c4801915af8ec1960f147, State: Initialized, Role: FOLLOWER
I20251025 08:01:54.148406 30450 raft_consensus.cc:740] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1b7a58056d4a0083000de1210e4d8f, State: Initialized, Role: FOLLOWER
I20251025 08:01:54.148520 30450 consensus_queue.cc:260] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.148520 30451 consensus_queue.cc:260] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.148785 30451 ts_tablet_manager.cc:1434] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 08:01:54.148785 30450 ts_tablet_manager.cc:1434] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 08:01:54.148741 30452 raft_consensus.cc:359] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.148864 30452 raft_consensus.cc:385] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:54.148900 30452 raft_consensus.cc:740] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9781542c5eaf4f68be13b04698643193, State: Initialized, Role: FOLLOWER
I20251025 08:01:54.149044 30452 consensus_queue.cc:260] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.149052 30456 tablet_replica.cc:442] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } } }
I20251025 08:01:54.149156 30456 tablet_replica.cc:445] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:54.149292 30452 ts_tablet_manager.cc:1434] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:01:54.149340 30457 tablet_replica.cc:442] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } } }
I20251025 08:01:54.149461 30457 tablet_replica.cc:445] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:54.150100 30458 tablet_replica.cc:442] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } } }
I20251025 08:01:54.150221 30458 tablet_replica.cc:445] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: This TxnStatusTablet replica's current role is: FOLLOWER
W20251025 08:01:54.222738 30428 tablet.cc:2378] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251025 08:01:54.239990 30286 tablet.cc:2378] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251025 08:01:54.248732 30144 tablet.cc:2378] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 08:01:54.420486 30458 raft_consensus.cc:493] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:01:54.420655 30458 raft_consensus.cc:515] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.421770 30458 leader_election.cc:290] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfb073222c6c4801915af8ec1960f147 (127.27.132.1:42569), cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623)
I20251025 08:01:54.424762 30472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.422075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:40623 (local address 127.27.132.2:41969)
1025 08:01:54.422356 (+ 281us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:54.422377 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:54.422484 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:54.422758 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:54.422761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:54.422780 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:54.422982 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.422988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.423756 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.423760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:54.424335 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.424343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.424464 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.424488 (+ 24us) client_negotiation.cc:769] Sending connection context
1025 08:01:54.424542 (+ 54us) client_negotiation.cc:240] Negotiation successful
1025 08:01:54.424603 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":93,"threads_started":1}
I20251025 08:01:54.425170 30449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.422225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:41969 (local address 127.27.132.3:40623)
1025 08:01:54.422350 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.422355 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.422443 (+ 88us) server_negotiation.cc:408] Connection header received
1025 08:01:54.422607 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.422611 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.422661 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.422733 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:54.423091 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.423644 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.424471 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.424924 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.424970 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.425024 (+ 54us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.425084 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 08:01:54.425583 30378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26efcb1a9b634e6c8131ac3a70359f14" candidate_uuid: "9781542c5eaf4f68be13b04698643193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1b7a58056d4a0083000de1210e4d8f" is_pre_election: true
I20251025 08:01:54.425755 30378 raft_consensus.cc:2468] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9781542c5eaf4f68be13b04698643193 in term 0.
I20251025 08:01:54.425802 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.422079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:42569 (local address 127.27.132.2:35089)
1025 08:01:54.422635 (+ 556us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:54.422647 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:54.422759 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:54.423015 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:54.423018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:54.423035 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:54.423249 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.423255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.424112 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.424117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:54.424770 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.424779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.425524 (+ 745us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.425542 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 08:01:54.425624 (+ 82us) client_negotiation.cc:240] Negotiation successful
1025 08:01:54.425681 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":278,"thread_start_us":93,"threads_started":1}
I20251025 08:01:54.425945 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.422180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:35089 (local address 127.27.132.1:42569)
1025 08:01:54.422327 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.422332 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.422740 (+ 408us) server_negotiation.cc:408] Connection header received
1025 08:01:54.422871 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.422874 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.422923 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.422993 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:54.423388 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.423911 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.425001 (+ 1090us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.425493 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.425525 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.425687 (+ 162us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.425778 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251025 08:01:54.426165 30170 leader_election.cc:304] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9781542c5eaf4f68be13b04698643193, cf1b7a58056d4a0083000de1210e4d8f; no voters:
I20251025 08:01:54.426440 30458 raft_consensus.cc:2804] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:01:54.426518 30458 raft_consensus.cc:493] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:01:54.426569 30458 raft_consensus.cc:3060] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:54.426533 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26efcb1a9b634e6c8131ac3a70359f14" candidate_uuid: "9781542c5eaf4f68be13b04698643193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb073222c6c4801915af8ec1960f147" is_pre_election: true
I20251025 08:01:54.426738 30097 raft_consensus.cc:2468] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9781542c5eaf4f68be13b04698643193 in term 0.
I20251025 08:01:54.427717 30458 raft_consensus.cc:515] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.428064 30458 leader_election.cc:290] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [CANDIDATE]: Term 1 election: Requested vote from peers cfb073222c6c4801915af8ec1960f147 (127.27.132.1:42569), cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623)
I20251025 08:01:54.428385 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26efcb1a9b634e6c8131ac3a70359f14" candidate_uuid: "9781542c5eaf4f68be13b04698643193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb073222c6c4801915af8ec1960f147"
I20251025 08:01:54.428483 30097 raft_consensus.cc:3060] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:54.428479 30378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26efcb1a9b634e6c8131ac3a70359f14" candidate_uuid: "9781542c5eaf4f68be13b04698643193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1b7a58056d4a0083000de1210e4d8f"
I20251025 08:01:54.428613 30378 raft_consensus.cc:3060] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:54.429466 30378 raft_consensus.cc:2468] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9781542c5eaf4f68be13b04698643193 in term 1.
I20251025 08:01:54.429737 30097 raft_consensus.cc:2468] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9781542c5eaf4f68be13b04698643193 in term 1.
I20251025 08:01:54.429769 30170 leader_election.cc:304] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9781542c5eaf4f68be13b04698643193, cf1b7a58056d4a0083000de1210e4d8f; no voters:
I20251025 08:01:54.429951 30458 raft_consensus.cc:2804] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:54.430155 30458 raft_consensus.cc:697] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [term 1 LEADER]: Becoming Leader. State: Replica: 9781542c5eaf4f68be13b04698643193, State: Running, Role: LEADER
I20251025 08:01:54.430512 30458 consensus_queue.cc:237] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.432390 30474 tablet_replica.cc:442] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: TxnStatusTablet state changed. Reason: New leader 9781542c5eaf4f68be13b04698643193. Latest consensus state: current_term: 1 leader_uuid: "9781542c5eaf4f68be13b04698643193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } } }
I20251025 08:01:54.432479 30474 tablet_replica.cc:445] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: This TxnStatusTablet replica's current role is: LEADER
I20251025 08:01:54.432736 30476 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251025 08:01:54.432916 30476 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251025 08:01:54.433872 29936 catalog_manager.cc:5649] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9781542c5eaf4f68be13b04698643193 (127.27.132.2). New cstate: current_term: 1 leader_uuid: "9781542c5eaf4f68be13b04698643193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } health_report { overall_health: UNKNOWN } } }
I20251025 08:01:54.445564 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.442581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.27.132.2:41373)
1025 08:01:54.442701 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.442704 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.442717 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:01:54.442787 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.442790 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.442842 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.442935 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:54.443360 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.443837 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.444482 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.444678 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.444737 (+ 59us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:54.445153 (+ 416us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:54.445260 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.445368 (+ 108us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.445420 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251025 08:01:54.445564 30468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.442356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:41373 (local address 127.0.0.1:53396)
1025 08:01:54.442536 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:54.442553 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:54.442679 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:54.442960 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:54.442963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:54.442972 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:01:54.443211 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.443217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.443950 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.443954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:54.444363 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.444371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.444492 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.444544 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:01:54.445261 (+ 717us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:01:54.445279 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 08:01:54.445351 (+ 72us) client_negotiation.cc:240] Negotiation successful
1025 08:01:54.445434 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20251025 08:01:54.450974 30378 raft_consensus.cc:1275] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Refusing update from remote peer 9781542c5eaf4f68be13b04698643193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:54.451118 30097 raft_consensus.cc:1275] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Refusing update from remote peer 9781542c5eaf4f68be13b04698643193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:54.451678 30458 consensus_queue.cc:1048] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:54.451869 30474 consensus_queue.cc:1048] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:54.455219 30457 tablet_replica.cc:442] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: TxnStatusTablet state changed. Reason: New leader 9781542c5eaf4f68be13b04698643193. Latest consensus state: current_term: 1 leader_uuid: "9781542c5eaf4f68be13b04698643193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } } }
I20251025 08:01:54.455389 30457 tablet_replica.cc:445] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:54.455256 30456 tablet_replica.cc:442] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: TxnStatusTablet state changed. Reason: New leader 9781542c5eaf4f68be13b04698643193. Latest consensus state: current_term: 1 leader_uuid: "9781542c5eaf4f68be13b04698643193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } } }
I20251025 08:01:54.456316 30456 tablet_replica.cc:445] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:54.458968 30458 tablet_replica.cc:442] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9781542c5eaf4f68be13b04698643193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } } }
I20251025 08:01:54.459020 30474 tablet_replica.cc:442] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9781542c5eaf4f68be13b04698643193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } } }
I20251025 08:01:54.459129 30474 tablet_replica.cc:445] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: This TxnStatusTablet replica's current role is: LEADER
I20251025 08:01:54.459988 30480 mvcc.cc:204] Tried to move back new op lower bound from 7214609671984218112 to 7214609671910010880. Current Snapshot: MvccSnapshot[applied={T|T < 7214609671984218112}]
I20251025 08:01:54.461269 30456 tablet_replica.cc:442] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9781542c5eaf4f68be13b04698643193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } } }
I20251025 08:01:54.461376 30456 tablet_replica.cc:445] T 26efcb1a9b634e6c8131ac3a70359f14 P cf1b7a58056d4a0083000de1210e4d8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:54.461357 30457 tablet_replica.cc:442] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9781542c5eaf4f68be13b04698643193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } } }
I20251025 08:01:54.461448 30457 tablet_replica.cc:445] T 26efcb1a9b634e6c8131ac3a70359f14 P cfb073222c6c4801915af8ec1960f147: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:01:54.459085 30458 tablet_replica.cc:445] T 26efcb1a9b634e6c8131ac3a70359f14 P 9781542c5eaf4f68be13b04698643193: This TxnStatusTablet replica's current role is: LEADER
I20251025 08:01:54.461814 30481 mvcc.cc:204] Tried to move back new op lower bound from 7214609671984218112 to 7214609671910010880. Current Snapshot: MvccSnapshot[applied={T|T < 7214609671984218112}]
I20251025 08:01:54.469238 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.465778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.27.132.1:42569)
1025 08:01:54.465923 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.465927 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.465942 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:01:54.466011 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.466014 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.466075 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.466173 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:54.466511 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.467318 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.468066 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.468268 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.468329 (+ 61us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:54.468863 (+ 534us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:54.468977 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.469057 (+ 80us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.469109 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251025 08:01:54.479475 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.476699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:41881 (local address 127.27.132.2:41373)
1025 08:01:54.476835 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.476840 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.477011 (+ 171us) server_negotiation.cc:408] Connection header received
1025 08:01:54.477165 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.477168 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.477216 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.477283 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:54.477667 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.478121 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.478786 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.478980 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.479027 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:54.479092 (+ 65us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:54.479158 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.479269 (+ 111us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.479313 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 08:01:54.479502 30491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.476594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:41373 (local address 127.27.132.1:41881)
1025 08:01:54.476925 (+ 331us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:54.476938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:54.477048 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:54.477306 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:54.477308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:54.477316 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:01:54.477559 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.477565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.478231 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.478234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:54.478674 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.478681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.478804 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.478849 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:01:54.479185 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:01:54.479190 (+ 5us) client_negotiation.cc:769] Sending connection context
1025 08:01:54.479253 (+ 63us) client_negotiation.cc:240] Negotiation successful
1025 08:01:54.479342 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20251025 08:01:54.487051 30378 raft_consensus.cc:1275] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Refusing update from remote peer cfb073222c6c4801915af8ec1960f147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:54.487167 30239 raft_consensus.cc:1275] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Refusing update from remote peer cfb073222c6c4801915af8ec1960f147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:54.487558 30461 consensus_queue.cc:1048] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:54.487710 30457 consensus_queue.cc:1048] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:54.512431 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.508332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:51969 (local address 127.27.132.1:42569)
1025 08:01:54.508451 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:54.508456 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:54.508955 (+ 499us) server_negotiation.cc:408] Connection header received
1025 08:01:54.509098 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:54.509102 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:54.509153 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:54.509221 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:54.509635 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.510110 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.511600 (+ 1490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.511797 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.512026 (+ 229us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:54.512098 (+ 72us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:54.512161 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:54.512253 (+ 92us) server_negotiation.cc:300] Negotiation successful
1025 08:01:54.512300 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 08:01:54.512455 30496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:54.507999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:42569 (local address 127.27.132.2:51969)
1025 08:01:54.508879 (+ 880us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:54.508895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:54.509003 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:54.509241 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:54.509244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:54.509252 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:01:54.509523 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.509529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.510222 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:54.510227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:54.511357 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:54.511369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:54.511869 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:54.511906 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:01:54.512179 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:01:54.512185 (+ 6us) client_negotiation.cc:769] Sending connection context
1025 08:01:54.512245 (+ 60us) client_negotiation.cc:240] Negotiation successful
1025 08:01:54.512317 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":97,"threads_started":1}
I20251025 08:01:54.540935 29938 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55708:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:54.541131 29938 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55708:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:54.543882 29938 catalog_manager.cc:5936] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069: Sending DeleteTablet for 3 replicas of tablet 893dc67cfa0c4583a34115614d7f6924
I20251025 08:01:54.544618 30077 tablet_service.cc:1552] Processing DeleteTablet for tablet 893dc67cfa0c4583a34115614d7f6924 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:54 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:36124
I20251025 08:01:54.545145 30219 tablet_service.cc:1552] Processing DeleteTablet for tablet 893dc67cfa0c4583a34115614d7f6924 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:54 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:53380
I20251025 08:01:54.545421 30361 tablet_service.cc:1552] Processing DeleteTablet for tablet 893dc67cfa0c4583a34115614d7f6924 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:54 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:55724
I20251025 08:01:54.545898 30501 tablet_replica.cc:333] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f: stopping tablet replica
I20251025 08:01:54.546170 30501 raft_consensus.cc:2243] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:01:54.546380 30501 raft_consensus.cc:2272] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:54.546959 29938 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:01:54.547473 29938 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:01:54.547726 30501 ts_tablet_manager.cc:1916] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:54.549530 30499 tablet_replica.cc:333] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147: stopping tablet replica
I20251025 08:01:54.549865 30499 raft_consensus.cc:2243] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:01:54.550307 30499 raft_consensus.cc:2272] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:54.551846 30499 ts_tablet_manager.cc:1916] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:54.553604 30219 tablet_service.cc:1505] Processing CreateTablet for tablet ed5da571fbe04e71b5878e1f2b8dacc9 (DEFAULT_TABLE table=test-table [id=3575f0291c2e45e4af3d0ebf79b3d940]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:54.553745 30361 tablet_service.cc:1505] Processing CreateTablet for tablet ed5da571fbe04e71b5878e1f2b8dacc9 (DEFAULT_TABLE table=test-table [id=3575f0291c2e45e4af3d0ebf79b3d940]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:54.553936 30219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5da571fbe04e71b5878e1f2b8dacc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:54.554136 30361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5da571fbe04e71b5878e1f2b8dacc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:54.555538 30500 tablet_replica.cc:333] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193: stopping tablet replica
I20251025 08:01:54.555795 30500 raft_consensus.cc:2243] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:01:54.555968 30500 raft_consensus.cc:2272] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:54.556465 30450 tablet_bootstrap.cc:492] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f: Bootstrap starting.
I20251025 08:01:54.553224 30077 tablet_service.cc:1505] Processing CreateTablet for tablet ed5da571fbe04e71b5878e1f2b8dacc9 (DEFAULT_TABLE table=test-table [id=3575f0291c2e45e4af3d0ebf79b3d940]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:54.556743 30077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5da571fbe04e71b5878e1f2b8dacc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:54.557199 30500 ts_tablet_manager.cc:1916] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:54.557435 30450 tablet_bootstrap.cc:654] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:54.558811 30451 tablet_bootstrap.cc:492] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147: Bootstrap starting.
I20251025 08:01:54.559422 30499 ts_tablet_manager.cc:1929] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:01:54.559592 30499 log.cc:1199] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/wal/wals/893dc67cfa0c4583a34115614d7f6924
I20251025 08:01:54.559839 30451 tablet_bootstrap.cc:654] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:54.559896 30500 ts_tablet_manager.cc:1929] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:01:54.559976 30500 log.cc:1199] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/893dc67cfa0c4583a34115614d7f6924
I20251025 08:01:54.559999 30452 tablet_bootstrap.cc:492] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193: Bootstrap starting.
I20251025 08:01:54.560181 30499 ts_tablet_manager.cc:1950] T 893dc67cfa0c4583a34115614d7f6924 P cfb073222c6c4801915af8ec1960f147: Deleting consensus metadata
I20251025 08:01:54.560284 30500 ts_tablet_manager.cc:1950] T 893dc67cfa0c4583a34115614d7f6924 P 9781542c5eaf4f68be13b04698643193: Deleting consensus metadata
I20251025 08:01:54.561026 30452 tablet_bootstrap.cc:654] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:54.561300 30501 ts_tablet_manager.cc:1929] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:01:54.561396 30501 log.cc:1199] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/893dc67cfa0c4583a34115614d7f6924
I20251025 08:01:54.561429 29922 catalog_manager.cc:4985] TS 9781542c5eaf4f68be13b04698643193 (127.27.132.2:41373): tablet 893dc67cfa0c4583a34115614d7f6924 (table test-table [id=e3bacd8db4014b69b4775b50d21fe148]) successfully deleted
I20251025 08:01:54.561739 30501 ts_tablet_manager.cc:1950] T 893dc67cfa0c4583a34115614d7f6924 P cf1b7a58056d4a0083000de1210e4d8f: Deleting consensus metadata
I20251025 08:01:54.562318 29925 catalog_manager.cc:4985] TS cfb073222c6c4801915af8ec1960f147 (127.27.132.1:42569): tablet 893dc67cfa0c4583a34115614d7f6924 (table test-table [id=e3bacd8db4014b69b4775b50d21fe148]) successfully deleted
I20251025 08:01:54.562739 29922 catalog_manager.cc:4985] TS cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623): tablet 893dc67cfa0c4583a34115614d7f6924 (table test-table [id=e3bacd8db4014b69b4775b50d21fe148]) successfully deleted
I20251025 08:01:54.563436 30452 tablet_bootstrap.cc:492] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193: No bootstrap required, opened a new log
I20251025 08:01:54.563527 30452 ts_tablet_manager.cc:1403] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251025 08:01:54.563625 30450 tablet_bootstrap.cc:492] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f: No bootstrap required, opened a new log
I20251025 08:01:54.563720 30450 ts_tablet_manager.cc:1403] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251025 08:01:54.564000 30452 raft_consensus.cc:359] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.564122 30452 raft_consensus.cc:385] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:54.564162 30452 raft_consensus.cc:740] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9781542c5eaf4f68be13b04698643193, State: Initialized, Role: FOLLOWER
I20251025 08:01:54.564114 30450 raft_consensus.cc:359] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.564234 30450 raft_consensus.cc:385] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:54.564272 30450 raft_consensus.cc:740] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1b7a58056d4a0083000de1210e4d8f, State: Initialized, Role: FOLLOWER
I20251025 08:01:54.564327 30452 consensus_queue.cc:260] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.564392 30450 consensus_queue.cc:260] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.564632 30452 ts_tablet_manager.cc:1434] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:01:54.564630 30450 ts_tablet_manager.cc:1434] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:01:54.566031 30451 tablet_bootstrap.cc:492] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147: No bootstrap required, opened a new log
I20251025 08:01:54.566133 30451 ts_tablet_manager.cc:1403] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251025 08:01:54.566608 30451 raft_consensus.cc:359] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.566768 30451 raft_consensus.cc:385] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:54.566814 30451 raft_consensus.cc:740] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb073222c6c4801915af8ec1960f147, State: Initialized, Role: FOLLOWER
I20251025 08:01:54.566962 30451 consensus_queue.cc:260] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.567232 30451 ts_tablet_manager.cc:1434] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:01:54.587203 30461 raft_consensus.cc:493] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:01:54.587369 30461 raft_consensus.cc:515] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.587800 30461 leader_election.cc:290] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9781542c5eaf4f68be13b04698643193 (127.27.132.2:41373), cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623)
I20251025 08:01:54.588171 30239 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed5da571fbe04e71b5878e1f2b8dacc9" candidate_uuid: "cfb073222c6c4801915af8ec1960f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9781542c5eaf4f68be13b04698643193" is_pre_election: true
I20251025 08:01:54.588244 30378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed5da571fbe04e71b5878e1f2b8dacc9" candidate_uuid: "cfb073222c6c4801915af8ec1960f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1b7a58056d4a0083000de1210e4d8f" is_pre_election: true
I20251025 08:01:54.588307 30239 raft_consensus.cc:2468] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfb073222c6c4801915af8ec1960f147 in term 0.
I20251025 08:01:54.588393 30378 raft_consensus.cc:2468] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfb073222c6c4801915af8ec1960f147 in term 0.
I20251025 08:01:54.588627 30028 leader_election.cc:304] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9781542c5eaf4f68be13b04698643193, cfb073222c6c4801915af8ec1960f147; no voters:
I20251025 08:01:54.588943 30461 raft_consensus.cc:2804] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:01:54.589010 30461 raft_consensus.cc:493] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:01:54.589047 30461 raft_consensus.cc:3060] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:54.590198 30461 raft_consensus.cc:515] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.590572 30461 leader_election.cc:290] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [CANDIDATE]: Term 1 election: Requested vote from peers 9781542c5eaf4f68be13b04698643193 (127.27.132.2:41373), cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623)
I20251025 08:01:54.590999 30239 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed5da571fbe04e71b5878e1f2b8dacc9" candidate_uuid: "cfb073222c6c4801915af8ec1960f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9781542c5eaf4f68be13b04698643193"
I20251025 08:01:54.591051 30378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed5da571fbe04e71b5878e1f2b8dacc9" candidate_uuid: "cfb073222c6c4801915af8ec1960f147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1b7a58056d4a0083000de1210e4d8f"
I20251025 08:01:54.591111 30239 raft_consensus.cc:3060] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:54.591156 30378 raft_consensus.cc:3060] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:54.592203 30378 raft_consensus.cc:2468] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfb073222c6c4801915af8ec1960f147 in term 1.
I20251025 08:01:54.592206 30239 raft_consensus.cc:2468] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfb073222c6c4801915af8ec1960f147 in term 1.
I20251025 08:01:54.592504 30028 leader_election.cc:304] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9781542c5eaf4f68be13b04698643193, cfb073222c6c4801915af8ec1960f147; no voters:
I20251025 08:01:54.592710 30461 raft_consensus.cc:2804] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:54.592806 30461 raft_consensus.cc:697] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 1 LEADER]: Becoming Leader. State: Replica: cfb073222c6c4801915af8ec1960f147, State: Running, Role: LEADER
I20251025 08:01:54.592937 30461 consensus_queue.cc:237] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:54.594580 29936 catalog_manager.cc:5649] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfb073222c6c4801915af8ec1960f147 (127.27.132.1). New cstate: current_term: 1 leader_uuid: "cfb073222c6c4801915af8ec1960f147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } health_report { overall_health: UNKNOWN } } }
I20251025 08:01:54.616784 30239 raft_consensus.cc:1275] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Refusing update from remote peer cfb073222c6c4801915af8ec1960f147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:54.617306 30378 raft_consensus.cc:1275] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Refusing update from remote peer cfb073222c6c4801915af8ec1960f147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:54.617269 30461 consensus_queue.cc:1048] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:54.617825 30461 consensus_queue.cc:1048] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 08:01:54.640300 30481 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251025 08:01:56.198827 29938 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55708:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:56.198972 29938 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55708:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:56.201382 29938 catalog_manager.cc:5936] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069: Sending DeleteTablet for 3 replicas of tablet ed5da571fbe04e71b5878e1f2b8dacc9
I20251025 08:01:56.201952 30077 tablet_service.cc:1552] Processing DeleteTablet for tablet ed5da571fbe04e71b5878e1f2b8dacc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:56 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:36124
I20251025 08:01:56.201983 30218 tablet_service.cc:1552] Processing DeleteTablet for tablet ed5da571fbe04e71b5878e1f2b8dacc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:56 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:53380
I20251025 08:01:56.202085 30361 tablet_service.cc:1552] Processing DeleteTablet for tablet ed5da571fbe04e71b5878e1f2b8dacc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:56 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:55724
I20251025 08:01:56.202334 30533 tablet_replica.cc:333] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147: stopping tablet replica
I20251025 08:01:56.202425 30534 tablet_replica.cc:333] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f: stopping tablet replica
I20251025 08:01:56.202481 30533 raft_consensus.cc:2243] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:01:56.202586 30534 raft_consensus.cc:2243] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:01:56.202711 30533 raft_consensus.cc:2272] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:56.202718 30534 raft_consensus.cc:2272] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:56.202988 30532 tablet_replica.cc:333] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193: stopping tablet replica
I20251025 08:01:56.203163 30532 raft_consensus.cc:2243] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:01:56.203348 30532 raft_consensus.cc:2272] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:56.203534 30533 ts_tablet_manager.cc:1916] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:56.203660 30534 ts_tablet_manager.cc:1916] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:56.204231 30532 ts_tablet_manager.cc:1916] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:56.206974 30533 ts_tablet_manager.cc:1929] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251025 08:01:56.207064 30533 log.cc:1199] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/wal/wals/ed5da571fbe04e71b5878e1f2b8dacc9
I20251025 08:01:56.207409 30533 ts_tablet_manager.cc:1950] T ed5da571fbe04e71b5878e1f2b8dacc9 P cfb073222c6c4801915af8ec1960f147: Deleting consensus metadata
I20251025 08:01:56.207916 30532 ts_tablet_manager.cc:1929] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251025 08:01:56.208020 30532 log.cc:1199] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/ed5da571fbe04e71b5878e1f2b8dacc9
I20251025 08:01:56.208316 30532 ts_tablet_manager.cc:1950] T ed5da571fbe04e71b5878e1f2b8dacc9 P 9781542c5eaf4f68be13b04698643193: Deleting consensus metadata
I20251025 08:01:56.208722 30534 ts_tablet_manager.cc:1929] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251025 08:01:56.208690 29925 catalog_manager.cc:4985] TS cfb073222c6c4801915af8ec1960f147 (127.27.132.1:42569): tablet ed5da571fbe04e71b5878e1f2b8dacc9 (table test-table [id=3575f0291c2e45e4af3d0ebf79b3d940]) successfully deleted
I20251025 08:01:56.208825 30534 log.cc:1199] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/ed5da571fbe04e71b5878e1f2b8dacc9
I20251025 08:01:56.208988 29922 catalog_manager.cc:4985] TS 9781542c5eaf4f68be13b04698643193 (127.27.132.2:41373): tablet ed5da571fbe04e71b5878e1f2b8dacc9 (table test-table [id=3575f0291c2e45e4af3d0ebf79b3d940]) successfully deleted
I20251025 08:01:56.209304 30534 ts_tablet_manager.cc:1950] T ed5da571fbe04e71b5878e1f2b8dacc9 P cf1b7a58056d4a0083000de1210e4d8f: Deleting consensus metadata
I20251025 08:01:56.209896 29922 catalog_manager.cc:4985] TS cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623): tablet ed5da571fbe04e71b5878e1f2b8dacc9 (table test-table [id=3575f0291c2e45e4af3d0ebf79b3d940]) successfully deleted
Oct 25 08:01:56 dist-test-slave-lq8l krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 25 08:01:56 dist-test-slave-lq8l krb5kdc[29890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:01:56.229372 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.220863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.27.132.62:46883)
1025 08:01:56.221124 (+ 261us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:56.221128 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:56.221143 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:01:56.221263 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:56.221265 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:56.221313 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:56.221393 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:01:56.222381 (+ 988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.222911 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.223763 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.223899 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.226324 (+ 2425us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:01:56.226345 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:01:56.226348 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:01:56.226373 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:01:56.227948 (+ 1575us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:56.228331 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:56.228335 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:56.228336 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:56.228377 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:01:56.228589 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:01:56.228592 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:01:56.228594 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:01:56.228753 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:01:56.228878 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:56.229082 (+ 204us) server_negotiation.cc:300] Negotiation successful
1025 08:01:56.229206 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
I20251025 08:01:56.231776 29938 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:01:56.232220 29938 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:01:56.236810 30077 tablet_service.cc:1505] Processing CreateTablet for tablet 35bca880194f4c92b864c7dda84d945e (DEFAULT_TABLE table=test-table [id=4eee176fbe0f465aa79ec085147a0504]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:56.236904 30218 tablet_service.cc:1505] Processing CreateTablet for tablet 35bca880194f4c92b864c7dda84d945e (DEFAULT_TABLE table=test-table [id=4eee176fbe0f465aa79ec085147a0504]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:56.237144 30077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35bca880194f4c92b864c7dda84d945e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:56.237119 30361 tablet_service.cc:1505] Processing CreateTablet for tablet 35bca880194f4c92b864c7dda84d945e (DEFAULT_TABLE table=test-table [id=4eee176fbe0f465aa79ec085147a0504]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:56.237226 30218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35bca880194f4c92b864c7dda84d945e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:56.237371 30361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35bca880194f4c92b864c7dda84d945e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:56.239302 30543 tablet_bootstrap.cc:492] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193: Bootstrap starting.
I20251025 08:01:56.239636 30544 tablet_bootstrap.cc:492] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147: Bootstrap starting.
I20251025 08:01:56.239755 30545 tablet_bootstrap.cc:492] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f: Bootstrap starting.
I20251025 08:01:56.240703 30545 tablet_bootstrap.cc:654] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:56.240756 30544 tablet_bootstrap.cc:654] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:56.241087 30543 tablet_bootstrap.cc:654] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:56.241930 30544 tablet_bootstrap.cc:492] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147: No bootstrap required, opened a new log
I20251025 08:01:56.241930 30545 tablet_bootstrap.cc:492] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f: No bootstrap required, opened a new log
I20251025 08:01:56.242020 30545 ts_tablet_manager.cc:1403] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 08:01:56.242020 30544 ts_tablet_manager.cc:1403] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 08:01:56.242447 30545 raft_consensus.cc:359] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.242456 30544 raft_consensus.cc:359] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.242565 30544 raft_consensus.cc:385] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:56.242565 30545 raft_consensus.cc:385] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:56.242609 30544 raft_consensus.cc:740] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb073222c6c4801915af8ec1960f147, State: Initialized, Role: FOLLOWER
I20251025 08:01:56.242609 30545 raft_consensus.cc:740] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1b7a58056d4a0083000de1210e4d8f, State: Initialized, Role: FOLLOWER
I20251025 08:01:56.242744 30545 consensus_queue.cc:260] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.242761 30544 consensus_queue.cc:260] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.243037 30544 ts_tablet_manager.cc:1434] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:01:56.243100 30545 ts_tablet_manager.cc:1434] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:01:56.244372 30543 tablet_bootstrap.cc:492] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193: No bootstrap required, opened a new log
I20251025 08:01:56.244462 30543 ts_tablet_manager.cc:1403] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20251025 08:01:56.244944 30543 raft_consensus.cc:359] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.245062 30543 raft_consensus.cc:385] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:56.245110 30543 raft_consensus.cc:740] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9781542c5eaf4f68be13b04698643193, State: Initialized, Role: FOLLOWER
I20251025 08:01:56.245252 30543 consensus_queue.cc:260] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.245534 30543 ts_tablet_manager.cc:1434] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 08:01:56.418877 30513 raft_consensus.cc:493] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:01:56.419031 30513 raft_consensus.cc:515] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.419554 30513 leader_election.cc:290] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfb073222c6c4801915af8ec1960f147 (127.27.132.1:42569), cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623)
I20251025 08:01:56.420011 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35bca880194f4c92b864c7dda84d945e" candidate_uuid: "9781542c5eaf4f68be13b04698643193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb073222c6c4801915af8ec1960f147" is_pre_election: true
I20251025 08:01:56.420161 30097 raft_consensus.cc:2468] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9781542c5eaf4f68be13b04698643193 in term 0.
I20251025 08:01:56.420114 30381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35bca880194f4c92b864c7dda84d945e" candidate_uuid: "9781542c5eaf4f68be13b04698643193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1b7a58056d4a0083000de1210e4d8f" is_pre_election: true
I20251025 08:01:56.420293 30381 raft_consensus.cc:2468] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9781542c5eaf4f68be13b04698643193 in term 0.
I20251025 08:01:56.420640 30173 leader_election.cc:304] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9781542c5eaf4f68be13b04698643193, cfb073222c6c4801915af8ec1960f147; no voters:
I20251025 08:01:56.420840 30513 raft_consensus.cc:2804] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:01:56.420900 30513 raft_consensus.cc:493] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:01:56.420928 30513 raft_consensus.cc:3060] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:56.421873 30513 raft_consensus.cc:515] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.422233 30513 leader_election.cc:290] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [CANDIDATE]: Term 1 election: Requested vote from peers cfb073222c6c4801915af8ec1960f147 (127.27.132.1:42569), cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623)
I20251025 08:01:56.422637 30097 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35bca880194f4c92b864c7dda84d945e" candidate_uuid: "9781542c5eaf4f68be13b04698643193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb073222c6c4801915af8ec1960f147"
I20251025 08:01:56.422637 30381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35bca880194f4c92b864c7dda84d945e" candidate_uuid: "9781542c5eaf4f68be13b04698643193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1b7a58056d4a0083000de1210e4d8f"
I20251025 08:01:56.422744 30381 raft_consensus.cc:3060] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:56.422758 30097 raft_consensus.cc:3060] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:56.423707 30381 raft_consensus.cc:2468] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9781542c5eaf4f68be13b04698643193 in term 1.
I20251025 08:01:56.423708 30097 raft_consensus.cc:2468] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9781542c5eaf4f68be13b04698643193 in term 1.
I20251025 08:01:56.424014 30173 leader_election.cc:304] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9781542c5eaf4f68be13b04698643193, cfb073222c6c4801915af8ec1960f147; no voters:
I20251025 08:01:56.424202 30513 raft_consensus.cc:2804] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:56.424291 30513 raft_consensus.cc:697] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 1 LEADER]: Becoming Leader. State: Replica: 9781542c5eaf4f68be13b04698643193, State: Running, Role: LEADER
I20251025 08:01:56.424443 30513 consensus_queue.cc:237] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.426112 29938 catalog_manager.cc:5649] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9781542c5eaf4f68be13b04698643193 (127.27.132.2). New cstate: current_term: 1 leader_uuid: "9781542c5eaf4f68be13b04698643193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } health_report { overall_health: UNKNOWN } } }
I20251025 08:01:56.484541 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.482167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.27.132.2:41373)
1025 08:01:56.482405 (+ 238us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:56.482409 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:56.482421 (+ 12us) server_negotiation.cc:408] Connection header received
1025 08:01:56.482470 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:56.482473 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:56.482517 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:56.482590 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:56.482873 (+ 283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.483385 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.483989 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.484120 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.484156 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:56.484222 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:56.484279 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:56.484365 (+ 86us) server_negotiation.cc:300] Negotiation successful
1025 08:01:56.484404 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
I20251025 08:01:56.492754 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.490209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:55297 (local address 127.27.132.2:41373)
1025 08:01:56.490313 (+ 104us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:56.490317 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:56.490493 (+ 176us) server_negotiation.cc:408] Connection header received
1025 08:01:56.490584 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:56.490587 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:56.490628 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:56.490701 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:56.491049 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.491537 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.492179 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.492304 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.492338 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:56.492401 (+ 63us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:56.492462 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:56.492580 (+ 118us) server_negotiation.cc:300] Negotiation successful
1025 08:01:56.492622 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251025 08:01:56.492762 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.490143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:41373 (local address 127.27.132.2:55297)
1025 08:01:56.490413 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:56.490426 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:56.490510 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:56.490743 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:56.490746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:56.490753 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:01:56.490948 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:56.490953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.491670 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.491673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:56.492080 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:56.492087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.492184 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.492207 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:01:56.492492 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:01:56.492497 (+ 5us) client_negotiation.cc:769] Sending connection context
1025 08:01:56.492554 (+ 57us) client_negotiation.cc:240] Negotiation successful
1025 08:01:56.492627 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20251025 08:01:56.498427 30381 raft_consensus.cc:1275] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Refusing update from remote peer 9781542c5eaf4f68be13b04698643193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:56.498806 30510 consensus_queue.cc:1048] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:56.502137 30096 raft_consensus.cc:1275] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Refusing update from remote peer 9781542c5eaf4f68be13b04698643193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:56.504372 30511 mvcc.cc:204] Tried to move back new op lower bound from 7214609680374534144 to 7214609680075751424. Current Snapshot: MvccSnapshot[applied={T|T < 7214609680374534144}]
I20251025 08:01:56.504578 30518 consensus_queue.cc:1048] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:56.535359 29936 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55738:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:56.535521 29936 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55738:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:56.537537 29936 catalog_manager.cc:5936] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069: Sending DeleteTablet for 3 replicas of tablet 35bca880194f4c92b864c7dda84d945e
I20251025 08:01:56.538169 30219 tablet_service.cc:1552] Processing DeleteTablet for tablet 35bca880194f4c92b864c7dda84d945e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:56 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:53380
I20251025 08:01:56.538168 30077 tablet_service.cc:1552] Processing DeleteTablet for tablet 35bca880194f4c92b864c7dda84d945e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:56 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:36124
I20251025 08:01:56.538259 30361 tablet_service.cc:1552] Processing DeleteTablet for tablet 35bca880194f4c92b864c7dda84d945e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:56 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:55724
I20251025 08:01:56.538393 30532 tablet_replica.cc:333] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193: stopping tablet replica
I20251025 08:01:56.538425 30534 tablet_replica.cc:333] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f: stopping tablet replica
I20251025 08:01:56.538515 30532 raft_consensus.cc:2243] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:01:56.538545 30534 raft_consensus.cc:2243] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:01:56.538600 30533 tablet_replica.cc:333] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147: stopping tablet replica
I20251025 08:01:56.538645 30534 raft_consensus.cc:2272] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:56.538727 30533 raft_consensus.cc:2243] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:01:56.538836 30532 raft_consensus.cc:2272] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:56.538843 30533 raft_consensus.cc:2272] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:56.539608 30532 ts_tablet_manager.cc:1916] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:56.539659 30533 ts_tablet_manager.cc:1916] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:56.541251 30534 ts_tablet_manager.cc:1916] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:56.542224 30533 ts_tablet_manager.cc:1929] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:01:56.542297 30533 log.cc:1199] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-0/wal/wals/35bca880194f4c92b864c7dda84d945e
I20251025 08:01:56.542407 30532 ts_tablet_manager.cc:1929] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:01:56.542469 30532 log.cc:1199] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/35bca880194f4c92b864c7dda84d945e
I20251025 08:01:56.542577 30533 ts_tablet_manager.cc:1950] T 35bca880194f4c92b864c7dda84d945e P cfb073222c6c4801915af8ec1960f147: Deleting consensus metadata
I20251025 08:01:56.542728 30532 ts_tablet_manager.cc:1950] T 35bca880194f4c92b864c7dda84d945e P 9781542c5eaf4f68be13b04698643193: Deleting consensus metadata
I20251025 08:01:56.543376 29925 catalog_manager.cc:4985] TS cfb073222c6c4801915af8ec1960f147 (127.27.132.1:42569): tablet 35bca880194f4c92b864c7dda84d945e (table test-table [id=4eee176fbe0f465aa79ec085147a0504]) successfully deleted
I20251025 08:01:56.543937 29922 catalog_manager.cc:4985] TS 9781542c5eaf4f68be13b04698643193 (127.27.132.2:41373): tablet 35bca880194f4c92b864c7dda84d945e (table test-table [id=4eee176fbe0f465aa79ec085147a0504]) successfully deleted
I20251025 08:01:56.544116 30534 ts_tablet_manager.cc:1929] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:01:56.544073 29936 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 08:01:56.544183 30534 log.cc:1199] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/35bca880194f4c92b864c7dda84d945e
I20251025 08:01:56.544438 30534 ts_tablet_manager.cc:1950] T 35bca880194f4c92b864c7dda84d945e P cf1b7a58056d4a0083000de1210e4d8f: Deleting consensus metadata
W20251025 08:01:56.544447 29936 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:01:56.545147 29922 catalog_manager.cc:4985] TS cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623): tablet 35bca880194f4c92b864c7dda84d945e (table test-table [id=4eee176fbe0f465aa79ec085147a0504]) successfully deleted
I20251025 08:01:56.548960 30077 tablet_service.cc:1505] Processing CreateTablet for tablet c4d333f764ff456d9a57dc6ae0efd490 (DEFAULT_TABLE table=test-table [id=0812a906404449899cc0f984beb151dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:56.549242 30077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d333f764ff456d9a57dc6ae0efd490. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:56.550916 30544 tablet_bootstrap.cc:492] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147: Bootstrap starting.
I20251025 08:01:56.551260 30219 tablet_service.cc:1505] Processing CreateTablet for tablet c4d333f764ff456d9a57dc6ae0efd490 (DEFAULT_TABLE table=test-table [id=0812a906404449899cc0f984beb151dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:56.551550 30219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d333f764ff456d9a57dc6ae0efd490. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:56.551890 30544 tablet_bootstrap.cc:654] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:56.552999 30544 tablet_bootstrap.cc:492] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147: No bootstrap required, opened a new log
I20251025 08:01:56.553092 30544 ts_tablet_manager.cc:1403] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251025 08:01:56.553562 30543 tablet_bootstrap.cc:492] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193: Bootstrap starting.
I20251025 08:01:56.553519 30544 raft_consensus.cc:359] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.553630 30544 raft_consensus.cc:385] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:56.553661 30544 raft_consensus.cc:740] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb073222c6c4801915af8ec1960f147, State: Initialized, Role: FOLLOWER
I20251025 08:01:56.553788 30544 consensus_queue.cc:260] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.554031 30544 ts_tablet_manager.cc:1434] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251025 08:01:56.554150 30361 tablet_service.cc:1505] Processing CreateTablet for tablet c4d333f764ff456d9a57dc6ae0efd490 (DEFAULT_TABLE table=test-table [id=0812a906404449899cc0f984beb151dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:01:56.554421 30361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d333f764ff456d9a57dc6ae0efd490. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:01:56.554476 30543 tablet_bootstrap.cc:654] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:56.556411 30545 tablet_bootstrap.cc:492] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f: Bootstrap starting.
I20251025 08:01:56.557389 30545 tablet_bootstrap.cc:654] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f: Neither blocks nor log segments found. Creating new log.
I20251025 08:01:56.557418 30543 tablet_bootstrap.cc:492] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193: No bootstrap required, opened a new log
I20251025 08:01:56.557514 30543 ts_tablet_manager.cc:1403] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20251025 08:01:56.557879 30543 raft_consensus.cc:359] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.557996 30543 raft_consensus.cc:385] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:56.558045 30543 raft_consensus.cc:740] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9781542c5eaf4f68be13b04698643193, State: Initialized, Role: FOLLOWER
I20251025 08:01:56.558176 30543 consensus_queue.cc:260] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.558491 30543 ts_tablet_manager.cc:1434] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251025 08:01:56.558722 30545 tablet_bootstrap.cc:492] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f: No bootstrap required, opened a new log
I20251025 08:01:56.558794 30545 ts_tablet_manager.cc:1403] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 08:01:56.559217 30545 raft_consensus.cc:359] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.559310 30545 raft_consensus.cc:385] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:01:56.559486 30545 raft_consensus.cc:740] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1b7a58056d4a0083000de1210e4d8f, State: Initialized, Role: FOLLOWER
I20251025 08:01:56.559608 30545 consensus_queue.cc:260] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.559883 30545 ts_tablet_manager.cc:1434] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251025 08:01:56.623407 30549 raft_consensus.cc:493] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:01:56.623566 30549 raft_consensus.cc:515] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.624646 30549 leader_election.cc:290] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfb073222c6c4801915af8ec1960f147 (127.27.132.1:42569), 9781542c5eaf4f68be13b04698643193 (127.27.132.2:41373)
I20251025 08:01:56.628118 30562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.624865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:42569 (local address 127.27.132.3:39965)
1025 08:01:56.625184 (+ 319us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:56.625203 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:56.625310 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:56.625777 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:56.625781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:56.625801 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:56.626096 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:56.626103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.627047 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.627051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:56.627680 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:56.627690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.627823 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.627850 (+ 27us) client_negotiation.cc:769] Sending connection context
1025 08:01:56.627905 (+ 55us) client_negotiation.cc:240] Negotiation successful
1025 08:01:56.627963 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":76,"threads_started":1}
I20251025 08:01:56.628710 30564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.625006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:39965 (local address 127.27.132.1:42569)
1025 08:01:56.625533 (+ 527us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:56.625537 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:56.625547 (+ 10us) server_negotiation.cc:408] Connection header received
1025 08:01:56.625590 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:56.625593 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:56.625647 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:56.625730 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:56.626226 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.626938 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.627807 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.628287 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.628324 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:56.628374 (+ 50us) server_negotiation.cc:300] Negotiation successful
1025 08:01:56.628426 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":89,"threads_started":1}
I20251025 08:01:56.628894 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.625282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:39497 (local address 127.27.132.2:41373)
1025 08:01:56.625409 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:56.625413 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:56.625427 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:01:56.625483 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:56.625487 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:56.625540 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:56.625629 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:01:56.626099 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.626796 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.628060 (+ 1264us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.628526 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.628563 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:56.628715 (+ 152us) server_negotiation.cc:300] Negotiation successful
1025 08:01:56.628773 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251025 08:01:56.629125 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.624903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:41373 (local address 127.27.132.3:39497)
1025 08:01:56.625259 (+ 356us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:56.625271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:56.625333 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:56.625662 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:56.625667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:56.625701 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:01:56.625964 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:56.625971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.626908 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.626911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:56.627910 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:56.627920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.628578 (+ 658us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.628597 (+ 19us) client_negotiation.cc:769] Sending connection context
1025 08:01:56.628669 (+ 72us) client_negotiation.cc:240] Negotiation successful
1025 08:01:56.628724 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20251025 08:01:56.629321 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4d333f764ff456d9a57dc6ae0efd490" candidate_uuid: "cf1b7a58056d4a0083000de1210e4d8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb073222c6c4801915af8ec1960f147" is_pre_election: true
I20251025 08:01:56.629480 30096 raft_consensus.cc:2468] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf1b7a58056d4a0083000de1210e4d8f in term 0.
I20251025 08:01:56.629534 30239 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4d333f764ff456d9a57dc6ae0efd490" candidate_uuid: "cf1b7a58056d4a0083000de1210e4d8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9781542c5eaf4f68be13b04698643193" is_pre_election: true
I20251025 08:01:56.629685 30239 raft_consensus.cc:2468] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf1b7a58056d4a0083000de1210e4d8f in term 0.
I20251025 08:01:56.629858 30315 leader_election.cc:304] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf1b7a58056d4a0083000de1210e4d8f, cfb073222c6c4801915af8ec1960f147; no voters:
I20251025 08:01:56.630102 30549 raft_consensus.cc:2804] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:01:56.630163 30549 raft_consensus.cc:493] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:01:56.630203 30549 raft_consensus.cc:3060] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:56.631294 30549 raft_consensus.cc:515] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.631724 30549 leader_election.cc:290] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [CANDIDATE]: Term 1 election: Requested vote from peers cfb073222c6c4801915af8ec1960f147 (127.27.132.1:42569), 9781542c5eaf4f68be13b04698643193 (127.27.132.2:41373)
I20251025 08:01:56.632011 30239 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4d333f764ff456d9a57dc6ae0efd490" candidate_uuid: "cf1b7a58056d4a0083000de1210e4d8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9781542c5eaf4f68be13b04698643193"
I20251025 08:01:56.632028 30096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4d333f764ff456d9a57dc6ae0efd490" candidate_uuid: "cf1b7a58056d4a0083000de1210e4d8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb073222c6c4801915af8ec1960f147"
I20251025 08:01:56.632123 30239 raft_consensus.cc:3060] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:56.632133 30096 raft_consensus.cc:3060] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:01:56.632936 30239 raft_consensus.cc:2468] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf1b7a58056d4a0083000de1210e4d8f in term 1.
I20251025 08:01:56.633158 30096 raft_consensus.cc:2468] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf1b7a58056d4a0083000de1210e4d8f in term 1.
I20251025 08:01:56.633220 30312 leader_election.cc:304] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9781542c5eaf4f68be13b04698643193, cf1b7a58056d4a0083000de1210e4d8f; no voters:
I20251025 08:01:56.633531 30549 raft_consensus.cc:2804] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:01:56.633760 30549 raft_consensus.cc:697] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 1 LEADER]: Becoming Leader. State: Replica: cf1b7a58056d4a0083000de1210e4d8f, State: Running, Role: LEADER
I20251025 08:01:56.634007 30549 consensus_queue.cc:237] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } }
I20251025 08:01:56.636348 29936 catalog_manager.cc:5649] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f reported cstate change: term changed from 0 to 1, leader changed from <none> to cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3). New cstate: current_term: 1 leader_uuid: "cf1b7a58056d4a0083000de1210e4d8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf1b7a58056d4a0083000de1210e4d8f" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40623 } health_report { overall_health: HEALTHY } } }
I20251025 08:01:56.666810 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.662973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.27.132.3:40623)
1025 08:01:56.663316 (+ 343us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:56.663321 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:56.663374 (+ 53us) server_negotiation.cc:408] Connection header received
1025 08:01:56.663466 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:56.663470 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:56.663543 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:56.663676 (+ 133us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:56.663988 (+ 312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.664864 (+ 876us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.665483 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.665702 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.665781 (+ 79us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:56.666407 (+ 626us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:56.666546 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:56.666580 (+ 34us) server_negotiation.cc:300] Negotiation successful
1025 08:01:56.666631 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":144,"threads_started":1}
I20251025 08:01:56.677641 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.674043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:41373 (local address 127.27.132.3:46563)
1025 08:01:56.674359 (+ 316us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:56.674379 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:56.674488 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:56.674790 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:56.674792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:56.674799 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:01:56.675024 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:56.675029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.676007 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.676010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:56.676817 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:56.676825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.676913 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.676951 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:01:56.677380 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:01:56.677383 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 08:01:56.677450 (+ 67us) client_negotiation.cc:240] Negotiation successful
1025 08:01:56.677511 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":121,"threads_started":1}
I20251025 08:01:56.677659 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.674232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:46563 (local address 127.27.132.2:41373)
1025 08:01:56.674419 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:56.674424 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:56.674442 (+ 18us) server_negotiation.cc:408] Connection header received
1025 08:01:56.674612 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:56.674616 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:56.674687 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:56.674790 (+ 103us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:56.675148 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.675903 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.676947 (+ 1044us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.677144 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.677199 (+ 55us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:56.677292 (+ 93us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:56.677381 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:56.677479 (+ 98us) server_negotiation.cc:300] Negotiation successful
1025 08:01:56.677528 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251025 08:01:56.685199 30239 raft_consensus.cc:1275] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Refusing update from remote peer cf1b7a58056d4a0083000de1210e4d8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:56.685739 30096 raft_consensus.cc:1275] T c4d333f764ff456d9a57dc6ae0efd490 P cfb073222c6c4801915af8ec1960f147 [term 1 FOLLOWER]: Refusing update from remote peer cf1b7a58056d4a0083000de1210e4d8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:01:56.685662 30565 consensus_queue.cc:1048] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9781542c5eaf4f68be13b04698643193" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:56.686151 30565 consensus_queue.cc:1048] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfb073222c6c4801915af8ec1960f147" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 42569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:01:56.704748 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.702115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:35083 (local address 127.27.132.3:40623)
1025 08:01:56.702241 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1025 08:01:56.702245 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:01:56.702259 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:01:56.702305 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:01:56.702308 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:01:56.702364 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:01:56.702448 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:01:56.702885 (+ 437us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.703426 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.704025 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.704228 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.704282 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:01:56.704367 (+ 85us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:01:56.704450 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1025 08:01:56.704574 (+ 124us) server_negotiation.cc:300] Negotiation successful
1025 08:01:56.704629 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251025 08:01:56.704772 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:56.701618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:40623 (local address 127.27.132.2:35083)
1025 08:01:56.701874 (+ 256us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:56.701892 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1025 08:01:56.701971 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:01:56.702484 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:01:56.702488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:01:56.702500 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:01:56.702758 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:56.702771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.703525 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:01:56.703529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:01:56.703920 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:01:56.703927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:01:56.704030 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:01:56.704062 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:01:56.704472 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:01:56.704479 (+ 7us) client_negotiation.cc:769] Sending connection context
1025 08:01:56.704554 (+ 75us) client_negotiation.cc:240] Negotiation successful
1025 08:01:56.704633 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
W20251025 08:01:56.705482 30515 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251025 08:01:58.264636 29938 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55738:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:58.264794 29938 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55738:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:01:58.267099 29938 catalog_manager.cc:5936] T 00000000000000000000000000000000 P fb647e644971498c88b4a76a0d6a4069: Sending DeleteTablet for 3 replicas of tablet c4d333f764ff456d9a57dc6ae0efd490
I20251025 08:01:58.267766 30219 tablet_service.cc:1552] Processing DeleteTablet for tablet c4d333f764ff456d9a57dc6ae0efd490 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:58 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:53380
I20251025 08:01:58.267827 30361 tablet_service.cc:1552] Processing DeleteTablet for tablet c4d333f764ff456d9a57dc6ae0efd490 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:58 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:55724
I20251025 08:01:58.268132 30077 tablet_service.cc:1552] Processing DeleteTablet for tablet c4d333f764ff456d9a57dc6ae0efd490 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:01:58 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:36124
I20251025 08:01:58.268177 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 30012
I20251025 08:01:58.270004 30596 tablet_replica.cc:333] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193: stopping tablet replica
I20251025 08:01:58.270164 30596 raft_consensus.cc:2243] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:01:58.270296 30596 raft_consensus.cc:2272] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:58.270558 30597 tablet_replica.cc:333] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f: stopping tablet replica
I20251025 08:01:58.270691 30597 raft_consensus.cc:2243] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:01:58.270983 30597 raft_consensus.cc:2272] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:01:58.271015 30596 ts_tablet_manager.cc:1916] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:58.271567 30597 ts_tablet_manager.cc:1916] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:01:58.273435 30597 ts_tablet_manager.cc:1929] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251025 08:01:58.273510 30597 log.cc:1199] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/c4d333f764ff456d9a57dc6ae0efd490
I20251025 08:01:58.273767 30597 ts_tablet_manager.cc:1950] T c4d333f764ff456d9a57dc6ae0efd490 P cf1b7a58056d4a0083000de1210e4d8f: Deleting consensus metadata
I20251025 08:01:58.274407 30596 ts_tablet_manager.cc:1929] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251025 08:01:58.274497 30596 log.cc:1199] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/c4d333f764ff456d9a57dc6ae0efd490
I20251025 08:01:58.274549 29922 catalog_manager.cc:4985] TS cf1b7a58056d4a0083000de1210e4d8f (127.27.132.3:40623): tablet c4d333f764ff456d9a57dc6ae0efd490 (table test-table [id=0812a906404449899cc0f984beb151dc]) successfully deleted
I20251025 08:01:58.274845 30596 ts_tablet_manager.cc:1950] T c4d333f764ff456d9a57dc6ae0efd490 P 9781542c5eaf4f68be13b04698643193: Deleting consensus metadata
I20251025 08:01:58.275645 29922 catalog_manager.cc:4985] TS 9781542c5eaf4f68be13b04698643193 (127.27.132.2:41373): tablet c4d333f764ff456d9a57dc6ae0efd490 (table test-table [id=0812a906404449899cc0f984beb151dc]) successfully deleted
W20251025 08:01:58.277101 29925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.132.1:42569 (error 108)
I20251025 08:01:58.277201 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 30154
I20251025 08:01:58.284413 30599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:01:58.283998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:42569 (local address 127.0.0.1:48416)
1025 08:01:58.284256 (+ 258us) negotiation.cc:107] Waiting for socket to connect
1025 08:01:58.284319 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.1:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
W20251025 08:01:58.284559 29925 catalog_manager.cc:4712] TS cfb073222c6c4801915af8ec1960f147 (127.27.132.1:42569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4d333f764ff456d9a57dc6ae0efd490: Network error: Client connection negotiation failed: client connection to 127.27.132.1:42569: connect: Connection refused (error 111)
I20251025 08:01:58.285418 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 30296
I20251025 08:01:58.292309 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 29906
2025-10-25T08:01:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7379 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:01:58 dist-test-slave-lq8l krb5kdc[30602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:01:58 dist-test-slave-lq8l krb5kdc[30602](info): set up 2 sockets
Oct 25 08:01:58 dist-test-slave-lq8l krb5kdc[30602](info): commencing operation
krb5kdc: starting...
W20251025 08:02:00.296324 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.981s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/test-user.keytab.
W20251025 08:02:00.495419 28176 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.140s user 0.001s sys 0.001s
Oct 25 08:02:00 dist-test-slave-lq8l krb5kdc[30602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:02:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
W20251025 08:02:01.551182 28176 mini_kdc.cc:279] Time spent creating service keytab for kudu/127.27.132.62: real 1.034s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
W20251025 08:02:02.451983 28176 mini_kdc.cc:279] Time spent creating service keytab for HTTP/127.27.132.62: real 0.901s user 0.000s sys 0.003s
I20251025 08:02:02.452525 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:34773
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:34773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:02.563915 30618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:02.564165 30618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:02.564225 30618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:02.567553 30618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:02.567629 30618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:02.567652 30618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:02.567673 30618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:02.567696 30618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:02.572237 30618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:34773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:34773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:02.573367 30618 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:02.574215 30618 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:02.580049 30623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:02.580117 30624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:02.580050 30626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:02.580403 30618 server_base.cc:1047] running on GCE node
I20251025 08:02:02.580910 30618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:02.581867 30618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:02.583037 30618 hybrid_clock.cc:648] HybridClock initialized: now 1761379322583007 us; error 45 us; skew 500 ppm
Oct 25 08:02:02 dist-test-slave-lq8l krb5kdc[30602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:02.585898 30618 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:02:02.587059 30618 webserver.cc:492] Webserver started at http://127.27.132.62:45781/ using document root <none> and password file <none>
I20251025 08:02:02.587699 30618 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:02.587759 30618 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:02.587956 30618 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:02.589936 30618 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "5a18065a8bd34a32a6804a348290e23a"
format_stamp: "Formatted at 2025-10-25 08:02:02 on dist-test-slave-lq8l"
server_key: "5073e5447e98674510429248ab7446c9"
server_key_iv: "871980a944dd1910cc94dab7fc430dd3"
server_key_version: "encryptionkey@0"
I20251025 08:02:02.590508 30618 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "5a18065a8bd34a32a6804a348290e23a"
format_stamp: "Formatted at 2025-10-25 08:02:02 on dist-test-slave-lq8l"
server_key: "5073e5447e98674510429248ab7446c9"
server_key_iv: "871980a944dd1910cc94dab7fc430dd3"
server_key_version: "encryptionkey@0"
I20251025 08:02:02.595001 30618 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20251025 08:02:02.597544 30633 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:02.598598 30618 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:02.598731 30618 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "5a18065a8bd34a32a6804a348290e23a"
format_stamp: "Formatted at 2025-10-25 08:02:02 on dist-test-slave-lq8l"
server_key: "5073e5447e98674510429248ab7446c9"
server_key_iv: "871980a944dd1910cc94dab7fc430dd3"
server_key_version: "encryptionkey@0"
I20251025 08:02:02.598836 30618 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:02.619805 30618 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:02.622831 30618 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:02.623031 30618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:02.630635 30618 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:34773
I20251025 08:02:02.630621 30685 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:34773 every 8 connection(s)
I20251025 08:02:02.631788 30618 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:02.634718 30686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:02.638569 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 30618
I20251025 08:02:02.638729 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:02.638975 28176 external_mini_cluster.cc:1468] Setting key 7a59cf6e54b24d6f3a68b862815e6ce3
I20251025 08:02:02.641062 30686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a: Bootstrap starting.
I20251025 08:02:02.643311 30686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:02.644119 30686 log.cc:826] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:02.646134 30686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a: No bootstrap required, opened a new log
Oct 25 08:02:02 dist-test-slave-lq8l krb5kdc[30602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:02.649562 30686 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18065a8bd34a32a6804a348290e23a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34773 } }
I20251025 08:02:02.649811 30686 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:02.649883 30686 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a18065a8bd34a32a6804a348290e23a, State: Initialized, Role: FOLLOWER
I20251025 08:02:02.650418 30686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18065a8bd34a32a6804a348290e23a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34773 } }
I20251025 08:02:02.650593 30686 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:02.650668 30686 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:02.650765 30686 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:02.652586 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:02.640470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.27.132.62:34773)
1025 08:02:02.640937 (+ 467us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:02.640948 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:02.640984 (+ 36us) server_negotiation.cc:408] Connection header received
1025 08:02:02.641577 (+ 593us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:02.641597 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:02.641928 (+ 331us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:02.642256 (+ 328us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:02.643231 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:02.644078 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:02.644792 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:02.645089 (+ 297us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:02.647666 (+ 2577us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:02.647687 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:02.647701 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:02.647754 (+ 53us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:02.649586 (+ 1832us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:02.650129 (+ 543us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:02.650138 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:02.650144 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:02.650257 (+ 113us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:02.650523 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:02.650525 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:02.650526 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:02.650978 (+ 452us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:02.651162 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:02.651469 (+ 307us) server_negotiation.cc:300] Negotiation successful
1025 08:02:02.651755 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":158,"threads_started":1}
I20251025 08:02:02.690757 30686 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18065a8bd34a32a6804a348290e23a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34773 } }
I20251025 08:02:02.691258 30686 leader_election.cc:304] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a18065a8bd34a32a6804a348290e23a; no voters:
I20251025 08:02:02.691680 30686 leader_election.cc:290] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:02.691969 30691 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:02.692286 30691 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [term 1 LEADER]: Becoming Leader. State: Replica: 5a18065a8bd34a32a6804a348290e23a, State: Running, Role: LEADER
I20251025 08:02:02.692642 30691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18065a8bd34a32a6804a348290e23a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34773 } }
I20251025 08:02:02.692998 30686 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:02.694422 30692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a18065a8bd34a32a6804a348290e23a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18065a8bd34a32a6804a348290e23a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34773 } } }
I20251025 08:02:02.694557 30692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:02.694752 30693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a18065a8bd34a32a6804a348290e23a. Latest consensus state: current_term: 1 leader_uuid: "5a18065a8bd34a32a6804a348290e23a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a18065a8bd34a32a6804a348290e23a" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34773 } } }
I20251025 08:02:02.694845 30693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:02.694953 30700 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:02.698314 30700 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:02.703665 30700 catalog_manager.cc:1357] Generated new cluster ID: 3ec9b8cec09547afb658d41266d75c1b
I20251025 08:02:02.703737 30700 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:02.721819 30700 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:02.722795 30700 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:02.731685 30700 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5a18065a8bd34a32a6804a348290e23a: Generated new TSK 0
I20251025 08:02:02.732605 30700 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
W20251025 08:02:04.065233 28176 mini_kdc.cc:279] Time spent creating service keytab for kudu/127.27.132.1: real 1.317s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:04.144766 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34773
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:04.252635 30714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:04.252856 30714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:04.252916 30714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:04.256126 30714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:04.256230 30714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:04.256352 30714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:02:04.260536 30714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:04.261584 30714 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:04.262400 30714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:04.269030 30720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:04.269023 30722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:04.269021 30719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:04.269068 30714 server_base.cc:1047] running on GCE node
I20251025 08:02:04.269564 30714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:04.270087 30714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:04.271229 30714 hybrid_clock.cc:648] HybridClock initialized: now 1761379324271219 us; error 26 us; skew 500 ppm
Oct 25 08:02:04 dist-test-slave-lq8l krb5kdc[30602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:04.273907 30714 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:02:04.274893 30714 webserver.cc:492] Webserver started at http://127.27.132.1:36709/ using document root <none> and password file <none>
I20251025 08:02:04.275477 30714 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:04.275537 30714 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:04.275733 30714 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:04.277431 30714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "8d75339c56f7437287732265d52be105"
format_stamp: "Formatted at 2025-10-25 08:02:04 on dist-test-slave-lq8l"
server_key: "aebba7fc45eacd05eb781fa81d7b9514"
server_key_iv: "a9f82fad27b196334ba5184261417108"
server_key_version: "encryptionkey@0"
I20251025 08:02:04.277904 30714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "8d75339c56f7437287732265d52be105"
format_stamp: "Formatted at 2025-10-25 08:02:04 on dist-test-slave-lq8l"
server_key: "aebba7fc45eacd05eb781fa81d7b9514"
server_key_iv: "a9f82fad27b196334ba5184261417108"
server_key_version: "encryptionkey@0"
I20251025 08:02:04.281212 30714 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:02:04.283275 30729 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:04.284278 30714 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:04.284384 30714 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "8d75339c56f7437287732265d52be105"
format_stamp: "Formatted at 2025-10-25 08:02:04 on dist-test-slave-lq8l"
server_key: "aebba7fc45eacd05eb781fa81d7b9514"
server_key_iv: "a9f82fad27b196334ba5184261417108"
server_key_version: "encryptionkey@0"
I20251025 08:02:04.284473 30714 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:04.297473 30714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:04.300477 30714 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:04.300669 30714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:04.301201 30714 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:04.302043 30714 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:04.302093 30714 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:04.302155 30714 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:04.302186 30714 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:04.311004 30714 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:44579
I20251025 08:02:04.311048 30842 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:44579 every 8 connection(s)
I20251025 08:02:04.311973 30714 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
Oct 25 08:02:04 dist-test-slave-lq8l krb5kdc[30602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:04.320740 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 30714
I20251025 08:02:04.320832 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:02:04.321055 28176 external_mini_cluster.cc:1468] Setting key 84918dd66fc0e72fc15235823751bf3e
I20251025 08:02:04.324815 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:04.313662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:34815 (local address 127.27.132.62:34773)
1025 08:02:04.313901 (+ 239us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:04.313904 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:04.314415 (+ 511us) server_negotiation.cc:408] Connection header received
1025 08:02:04.315268 (+ 853us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:04.315272 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:04.315364 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:04.315452 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:04.317467 (+ 2015us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.318089 (+ 622us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:04.318711 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.318843 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:04.321196 (+ 2353us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:04.321219 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:04.321223 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:04.321256 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:04.322841 (+ 1585us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:04.323355 (+ 514us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:04.323363 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:04.323367 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:04.323421 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:04.323703 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:04.323707 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:04.323708 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:04.323867 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:04.323964 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:04.324494 (+ 530us) server_negotiation.cc:300] Negotiation successful
1025 08:02:04.324624 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20251025 08:02:04.325464 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:04.313875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34773 (local address 127.27.132.1:34815)
1025 08:02:04.314280 (+ 405us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:04.314315 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:04.315059 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:04.315631 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:04.315639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:04.316033 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:04.317287 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:04.317308 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.318201 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:04.318204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:04.318619 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:04.318626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.318812 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:04.319481 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:04.319506 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:04.321025 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:04.322965 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:04.322971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:04.322984 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:04.323223 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:04.323513 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:04.323516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:04.323518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:04.323609 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:04.323986 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:04.323993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:04.324243 (+ 250us) client_negotiation.cc:769] Sending connection context
1025 08:02:04.324439 (+ 196us) client_negotiation.cc:240] Negotiation successful
1025 08:02:04.324729 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20251025 08:02:04.326664 30843 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34773
I20251025 08:02:04.326969 30843 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:04.327682 30843 heartbeater.cc:507] Master 127.27.132.62:34773 requested a full tablet report, sending...
I20251025 08:02:04.329382 30650 ts_manager.cc:194] Registered new tserver with Master: 8d75339c56f7437287732265d52be105 (127.27.132.1:44579)
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
I20251025 08:02:04.330771 30650 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:34815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:04.363920 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34773
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:04.469250 30851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:04.469489 30851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:04.469585 30851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:04.472811 30851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:04.472885 30851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:04.473006 30851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:02:04.477111 30851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:04.478154 30851 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:04.478993 30851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:04.485412 30856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:04.485412 30859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:04.485608 30857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:04.485476 30851 server_base.cc:1047] running on GCE node
I20251025 08:02:04.486032 30851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:04.486580 30851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:04.487725 30851 hybrid_clock.cc:648] HybridClock initialized: now 1761379324487718 us; error 28 us; skew 500 ppm
Oct 25 08:02:04 dist-test-slave-lq8l krb5kdc[30602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:04.490381 30851 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:02:04.491422 30851 webserver.cc:492] Webserver started at http://127.27.132.2:40005/ using document root <none> and password file <none>
I20251025 08:02:04.491976 30851 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:04.492033 30851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:04.492216 30851 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:04.493975 30851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "80862d9d6a524e63989b8c708f95523b"
format_stamp: "Formatted at 2025-10-25 08:02:04 on dist-test-slave-lq8l"
server_key: "3105d375225f48b3721064dd3a888fe9"
server_key_iv: "40faaaeee3448884d5b0515b0f276d68"
server_key_version: "encryptionkey@0"
I20251025 08:02:04.494462 30851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "80862d9d6a524e63989b8c708f95523b"
format_stamp: "Formatted at 2025-10-25 08:02:04 on dist-test-slave-lq8l"
server_key: "3105d375225f48b3721064dd3a888fe9"
server_key_iv: "40faaaeee3448884d5b0515b0f276d68"
server_key_version: "encryptionkey@0"
I20251025 08:02:04.498009 30851 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251025 08:02:04.500344 30866 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:04.501334 30851 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:04.501467 30851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "80862d9d6a524e63989b8c708f95523b"
format_stamp: "Formatted at 2025-10-25 08:02:04 on dist-test-slave-lq8l"
server_key: "3105d375225f48b3721064dd3a888fe9"
server_key_iv: "40faaaeee3448884d5b0515b0f276d68"
server_key_version: "encryptionkey@0"
I20251025 08:02:04.501564 30851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:04.515605 30851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:04.518558 30851 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:04.518775 30851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:04.519380 30851 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:04.520236 30851 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:04.520287 30851 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:04.520346 30851 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:04.520380 30851 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:04.528863 30851 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:41299
I20251025 08:02:04.528903 30979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:41299 every 8 connection(s)
I20251025 08:02:04.529815 30851 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
Oct 25 08:02:04 dist-test-slave-lq8l krb5kdc[30602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:04.539750 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 30851
I20251025 08:02:04.539878 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:02:04.540166 28176 external_mini_cluster.cc:1468] Setting key 1b2ff95f08756299583a4ef710a2a5c3
I20251025 08:02:04.542053 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:04.531633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:34901 (local address 127.27.132.62:34773)
1025 08:02:04.531804 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:04.531809 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:04.532486 (+ 677us) server_negotiation.cc:408] Connection header received
1025 08:02:04.533259 (+ 773us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:04.533262 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:04.533311 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:04.533387 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:04.535006 (+ 1619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.535574 (+ 568us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:04.536168 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.536307 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:04.538370 (+ 2063us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:04.538391 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:04.538394 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:04.538435 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:04.540057 (+ 1622us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:04.540547 (+ 490us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:04.540552 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:04.540553 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:04.540604 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:04.540886 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:04.540891 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:04.540893 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:04.541044 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:04.541125 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:04.541741 (+ 616us) server_negotiation.cc:300] Negotiation successful
1025 08:02:04.541867 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251025 08:02:04.542665 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:04.531925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34773 (local address 127.27.132.2:34901)
1025 08:02:04.532339 (+ 414us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:04.532375 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:04.533085 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:04.533529 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:04.533537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:04.533932 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:04.534864 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:04.534878 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.535664 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:04.535667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:04.536074 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:04.536081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.536249 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:04.536825 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:04.536847 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:04.538223 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:04.540174 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:04.540179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:04.540188 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:04.540445 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:04.540694 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:04.540697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:04.540699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:04.540800 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:04.541146 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:04.541154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:04.541446 (+ 292us) client_negotiation.cc:769] Sending connection context
1025 08:02:04.541684 (+ 238us) client_negotiation.cc:240] Negotiation successful
1025 08:02:04.541920 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20251025 08:02:04.543921 30980 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34773
I20251025 08:02:04.544159 30980 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:04.544663 30980 heartbeater.cc:507] Master 127.27.132.62:34773 requested a full tablet report, sending...
I20251025 08:02:04.545693 30650 ts_manager.cc:194] Registered new tserver with Master: 80862d9d6a524e63989b8c708f95523b (127.27.132.2:41299)
I20251025 08:02:04.546237 30650 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:34901
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:04.586077 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34773
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:04.693881 30987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:04.694095 30987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:04.694154 30987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:04.697365 30987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:04.697433 30987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:04.697515 30987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:02:04.701673 30987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:04.702685 30987 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:04.703584 30987 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:04.709950 30992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:04.709942 30995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:04.710038 30987 server_base.cc:1047] running on GCE node
W20251025 08:02:04.709950 30993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:04.710533 30987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:04.711042 30987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:04.712188 30987 hybrid_clock.cc:648] HybridClock initialized: now 1761379324712174 us; error 30 us; skew 500 ppm
Oct 25 08:02:04 dist-test-slave-lq8l krb5kdc[30602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:04.714756 30987 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:02:04.715760 30987 webserver.cc:492] Webserver started at http://127.27.132.3:40991/ using document root <none> and password file <none>
I20251025 08:02:04.716311 30987 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:04.716361 30987 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:04.716521 30987 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:04.718127 30987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "605d208db69c4bf49758ae953626202e"
format_stamp: "Formatted at 2025-10-25 08:02:04 on dist-test-slave-lq8l"
server_key: "152767fe14c3e8b23bbe6edb3eeae800"
server_key_iv: "790c4d944d8e9152ca6382dcd0cfc9d2"
server_key_version: "encryptionkey@0"
I20251025 08:02:04.718549 30987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "605d208db69c4bf49758ae953626202e"
format_stamp: "Formatted at 2025-10-25 08:02:04 on dist-test-slave-lq8l"
server_key: "152767fe14c3e8b23bbe6edb3eeae800"
server_key_iv: "790c4d944d8e9152ca6382dcd0cfc9d2"
server_key_version: "encryptionkey@0"
I20251025 08:02:04.721823 30987 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251025 08:02:04.723902 31002 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:04.724891 30987 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:04.725001 30987 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "605d208db69c4bf49758ae953626202e"
format_stamp: "Formatted at 2025-10-25 08:02:04 on dist-test-slave-lq8l"
server_key: "152767fe14c3e8b23bbe6edb3eeae800"
server_key_iv: "790c4d944d8e9152ca6382dcd0cfc9d2"
server_key_version: "encryptionkey@0"
I20251025 08:02:04.725090 30987 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:04.742386 30987 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:04.745432 30987 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:04.745630 30987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:04.746186 30987 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:04.747078 30987 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:04.747133 30987 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:04.747196 30987 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:04.747225 30987 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:04.756289 30987 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:41217
I20251025 08:02:04.756317 31115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:41217 every 8 connection(s)
I20251025 08:02:04.757261 30987 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:02:04.761809 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 30987
I20251025 08:02:04.761935 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:02:04.762221 28176 external_mini_cluster.cc:1468] Setting key 3f0d4dd43ee9c298119444f114c0c22a
Oct 25 08:02:04 dist-test-slave-lq8l krb5kdc[30602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:04.769394 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:04.759067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:52807 (local address 127.27.132.62:34773)
1025 08:02:04.759218 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:04.759223 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:04.759912 (+ 689us) server_negotiation.cc:408] Connection header received
1025 08:02:04.760766 (+ 854us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:04.760769 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:04.760834 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:04.760919 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:04.762914 (+ 1995us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.763511 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:04.764174 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.764359 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:04.766460 (+ 2101us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:04.766476 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:04.766478 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:04.766505 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:04.767795 (+ 1290us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:04.768235 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:04.768238 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:04.768239 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:04.768283 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:04.768517 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:04.768519 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:04.768520 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:04.768660 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:04.768741 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:04.769166 (+ 425us) server_negotiation.cc:300] Negotiation successful
1025 08:02:04.769265 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251025 08:02:04.770056 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:04.759348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34773 (local address 127.27.132.3:52807)
1025 08:02:04.759777 (+ 429us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:04.759813 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:04.760563 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:04.761094 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:04.761102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:04.761510 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:04.762762 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:04.762777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.763636 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:04.763639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:04.764052 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:04.764059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:04.764282 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:04.764897 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:04.764920 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:04.766325 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:04.767901 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:04.767905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:04.767914 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:04.768144 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:04.768357 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:04.768360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:04.768361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:04.768446 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:04.768737 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:04.768741 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:04.768967 (+ 226us) client_negotiation.cc:769] Sending connection context
1025 08:02:04.769141 (+ 174us) client_negotiation.cc:240] Negotiation successful
1025 08:02:04.769356 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20251025 08:02:04.771142 31116 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34773
I20251025 08:02:04.771422 31116 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:04.771950 31116 heartbeater.cc:507] Master 127.27.132.62:34773 requested a full tablet report, sending...
I20251025 08:02:04.772928 30650 ts_manager.cc:194] Registered new tserver with Master: 605d208db69c4bf49758ae953626202e (127.27.132.3:41217)
I20251025 08:02:04.773492 30650 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:52807
I20251025 08:02:04.775879 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251025 08:02:04.787628 30845 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:04.786245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.27.132.62:34773)
1025 08:02:04.786410 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:04.786416 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:04.786503 (+ 87us) server_negotiation.cc:408] Connection header received
1025 08:02:04.786641 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:04.786644 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:04.786696 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:04.786769 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:04.787494 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56226: BlockingRecv error: recv got EOF from 127.0.0.1:56226 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20251025 08:02:04.788431 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 30714
I20251025 08:02:04.794154 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 30851
I20251025 08:02:04.799468 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 30987
I20251025 08:02:04.804886 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 30618
2025-10-25T08:02:04Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (6511 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:02:04 dist-test-slave-lq8l krb5kdc[31128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:02:04 dist-test-slave-lq8l krb5kdc[31128](info): set up 2 sockets
Oct 25 08:02:04 dist-test-slave-lq8l krb5kdc[31128](info): commencing operation
krb5kdc: starting...
W20251025 08:02:06.792490 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.965s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:02:06 dist-test-slave-lq8l krb5kdc[31128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:02:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:06.932246 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:44849
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:44849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:07.042310 31144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:07.042585 31144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:07.042682 31144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:07.045992 31144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:07.046069 31144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:07.046108 31144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:07.046141 31144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:07.046181 31144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:07.050565 31144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:44849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:44849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:07.051743 31144 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:07.052615 31144 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:07.058437 31149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:07.058426 31152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:07.058425 31150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:07.058727 31144 server_base.cc:1047] running on GCE node
I20251025 08:02:07.059423 31144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:07.060377 31144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:07.061570 31144 hybrid_clock.cc:648] HybridClock initialized: now 1761379327061549 us; error 40 us; skew 500 ppm
Oct 25 08:02:07 dist-test-slave-lq8l krb5kdc[31128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:07.064378 31144 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:02:07.065431 31144 webserver.cc:492] Webserver started at http://127.27.132.62:39067/ using document root <none> and password file <none>
I20251025 08:02:07.065991 31144 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:07.066049 31144 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:07.066236 31144 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:07.067997 31144 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "eaac9724540b4695be5e848f77819c8f"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "45c4605f573a956fd4da346da1500f40"
server_key_iv: "7660ec280d3cd34582cb4da005377972"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.068468 31144 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "eaac9724540b4695be5e848f77819c8f"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "45c4605f573a956fd4da346da1500f40"
server_key_iv: "7660ec280d3cd34582cb4da005377972"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.071813 31144 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:02:07.074061 31159 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:07.075053 31144 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:07.075167 31144 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "eaac9724540b4695be5e848f77819c8f"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "45c4605f573a956fd4da346da1500f40"
server_key_iv: "7660ec280d3cd34582cb4da005377972"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.075256 31144 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:07.088882 31144 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:07.092048 31144 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:07.092234 31144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:07.099828 31211 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:44849 every 8 connection(s)
I20251025 08:02:07.099824 31144 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:44849
I20251025 08:02:07.101078 31144 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:07.104027 31212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:07.108373 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 31144
I20251025 08:02:07.108489 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:07.108728 28176 external_mini_cluster.cc:1468] Setting key 6fee4a757d10bf45fef01e478b7a256a
I20251025 08:02:07.109860 31212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f: Bootstrap starting.
I20251025 08:02:07.112538 31212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:07.113353 31212 log.cc:826] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:07.115283 31212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f: No bootstrap required, opened a new log
Oct 25 08:02:07 dist-test-slave-lq8l krb5kdc[31128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:07.118450 31212 raft_consensus.cc:359] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaac9724540b4695be5e848f77819c8f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44849 } }
I20251025 08:02:07.118685 31212 raft_consensus.cc:385] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:07.118757 31212 raft_consensus.cc:740] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaac9724540b4695be5e848f77819c8f, State: Initialized, Role: FOLLOWER
I20251025 08:02:07.119226 31212 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaac9724540b4695be5e848f77819c8f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44849 } }
I20251025 08:02:07.119401 31212 raft_consensus.cc:399] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:07.119470 31212 raft_consensus.cc:493] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:07.119575 31212 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:07.120643 31212 raft_consensus.cc:515] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaac9724540b4695be5e848f77819c8f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44849 } }
I20251025 08:02:07.121038 31212 leader_election.cc:304] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaac9724540b4695be5e848f77819c8f; no voters:
I20251025 08:02:07.121369 31212 leader_election.cc:290] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:07.121508 31217 raft_consensus.cc:2804] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:07.121718 31217 raft_consensus.cc:697] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [term 1 LEADER]: Becoming Leader. State: Replica: eaac9724540b4695be5e848f77819c8f, State: Running, Role: LEADER
I20251025 08:02:07.122059 31217 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaac9724540b4695be5e848f77819c8f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44849 } }
I20251025 08:02:07.122416 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:07.110069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.27.132.62:44849)
1025 08:02:07.110540 (+ 471us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:07.110549 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:07.110581 (+ 32us) server_negotiation.cc:408] Connection header received
1025 08:02:07.111135 (+ 554us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:07.111152 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:07.111485 (+ 333us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:07.111795 (+ 310us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:07.113278 (+ 1483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.114083 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:07.114709 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.115378 (+ 669us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:07.117272 (+ 1894us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:07.117294 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:07.117308 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:07.117346 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:07.119444 (+ 2098us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:07.119892 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:07.119898 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:07.119904 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:07.119989 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:07.120162 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:07.120165 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:07.120167 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:07.120601 (+ 434us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:07.120824 (+ 223us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:07.121109 (+ 285us) server_negotiation.cc:300] Negotiation successful
1025 08:02:07.121385 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":104,"threads_started":1}
I20251025 08:02:07.122578 31212 sys_catalog.cc:565] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:07.123894 31218 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaac9724540b4695be5e848f77819c8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaac9724540b4695be5e848f77819c8f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44849 } } }
I20251025 08:02:07.124006 31218 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:07.124363 31226 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:07.124850 31219 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaac9724540b4695be5e848f77819c8f. Latest consensus state: current_term: 1 leader_uuid: "eaac9724540b4695be5e848f77819c8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaac9724540b4695be5e848f77819c8f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44849 } } }
I20251025 08:02:07.124943 31219 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:07.126973 31226 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:07.132385 31226 catalog_manager.cc:1357] Generated new cluster ID: 8a5fa938891542d99a5d2195ed8cf66a
I20251025 08:02:07.132474 31226 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:07.147821 31226 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:07.149209 31226 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:07.158634 31226 catalog_manager.cc:6022] T 00000000000000000000000000000000 P eaac9724540b4695be5e848f77819c8f: Generated new TSK 0
I20251025 08:02:07.159435 31226 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:07.214320 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:44849
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:07.326570 31240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:07.326828 31240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:07.326913 31240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:07.330258 31240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:07.330327 31240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:07.330425 31240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:02:07.334636 31240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:44849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:07.335768 31240 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:07.336581 31240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:07.342988 31246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:07.342988 31248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:07.343000 31245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:07.343132 31240 server_base.cc:1047] running on GCE node
I20251025 08:02:07.343731 31240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:07.344300 31240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:07.345465 31240 hybrid_clock.cc:648] HybridClock initialized: now 1761379327345419 us; error 70 us; skew 500 ppm
Oct 25 08:02:07 dist-test-slave-lq8l krb5kdc[31128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:07.348268 31240 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:02:07.349268 31240 webserver.cc:492] Webserver started at http://127.27.132.1:36247/ using document root <none> and password file <none>
I20251025 08:02:07.349843 31240 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:07.349895 31240 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:07.350056 31240 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:07.351804 31240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "b9093e0b3ab341b09fc9a1b1514c1f93"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "2967742def10509579e0b10fdd13d07d"
server_key_iv: "75d547875c9eeccb5296c269d1e6baf6"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.352231 31240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "b9093e0b3ab341b09fc9a1b1514c1f93"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "2967742def10509579e0b10fdd13d07d"
server_key_iv: "75d547875c9eeccb5296c269d1e6baf6"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.355506 31240 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251025 08:02:07.357641 31255 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:07.358664 31240 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:07.358779 31240 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "b9093e0b3ab341b09fc9a1b1514c1f93"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "2967742def10509579e0b10fdd13d07d"
server_key_iv: "75d547875c9eeccb5296c269d1e6baf6"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.358868 31240 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:07.370919 31240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:07.374235 31240 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:07.374428 31240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:07.374975 31240 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:07.375871 31240 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:07.375922 31240 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:07.375981 31240 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:07.376010 31240 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:07.384655 31240 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:42493
I20251025 08:02:07.384692 31368 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:42493 every 8 connection(s)
I20251025 08:02:07.385614 31240 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:02:07.390422 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 31240
I20251025 08:02:07.390523 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:02:07.390755 28176 external_mini_cluster.cc:1468] Setting key 034d5e07c53a7abf53ca9b25f739fa57
Oct 25 08:02:07 dist-test-slave-lq8l krb5kdc[31128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:07.399664 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:07.387462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:44585 (local address 127.27.132.62:44849)
1025 08:02:07.387627 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:07.387632 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:07.388310 (+ 678us) server_negotiation.cc:408] Connection header received
1025 08:02:07.389135 (+ 825us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:07.389138 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:07.389189 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:07.389267 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:07.390695 (+ 1428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.391390 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:07.392039 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.392219 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:07.395439 (+ 3220us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:07.395464 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:07.395467 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:07.395498 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:07.397559 (+ 2061us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:07.398140 (+ 581us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:07.398143 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:07.398145 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:07.398195 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:07.398485 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:07.398489 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:07.398491 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:07.398663 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:07.398757 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:07.399299 (+ 542us) server_negotiation.cc:300] Negotiation successful
1025 08:02:07.399489 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
I20251025 08:02:07.400362 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:07.387689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:44849 (local address 127.27.132.1:44585)
1025 08:02:07.388158 (+ 469us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:07.388196 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:07.388950 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:07.389430 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:07.389438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:07.389855 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:07.390466 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:07.390481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.391519 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:07.391523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:07.391918 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:07.391925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.392586 (+ 661us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:07.393223 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:07.393251 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:07.395234 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:07.397736 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:07.397742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:07.397754 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:07.398022 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:07.398291 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:07.398294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:07.398296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:07.398392 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:07.398768 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:07.398778 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:07.399048 (+ 270us) client_negotiation.cc:769] Sending connection context
1025 08:02:07.399253 (+ 205us) client_negotiation.cc:240] Negotiation successful
1025 08:02:07.399580 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":111,"threads_started":1}
I20251025 08:02:07.401643 31369 heartbeater.cc:344] Connected to a master server at 127.27.132.62:44849
I20251025 08:02:07.401904 31369 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
I20251025 08:02:07.402462 31369 heartbeater.cc:507] Master 127.27.132.62:44849 requested a full tablet report, sending...
I20251025 08:02:07.404071 31176 ts_manager.cc:194] Registered new tserver with Master: b9093e0b3ab341b09fc9a1b1514c1f93 (127.27.132.1:42493)
I20251025 08:02:07.405342 31176 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:44585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:07.439682 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:44849
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:07.548197 31376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:07.548457 31376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:07.548519 31376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:07.552052 31376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:07.552117 31376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:07.552217 31376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:02:07.556568 31376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:44849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:07.557634 31376 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:07.558481 31376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:07.564975 31384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:07.565058 31381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:07.565093 31376 server_base.cc:1047] running on GCE node
W20251025 08:02:07.564975 31382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:07.565625 31376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:07.566182 31376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:07.567365 31376 hybrid_clock.cc:648] HybridClock initialized: now 1761379327567343 us; error 37 us; skew 500 ppm
Oct 25 08:02:07 dist-test-slave-lq8l krb5kdc[31128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:07.570168 31376 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:02:07.571250 31376 webserver.cc:492] Webserver started at http://127.27.132.2:34735/ using document root <none> and password file <none>
I20251025 08:02:07.571834 31376 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:07.571885 31376 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:07.572046 31376 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:07.573745 31376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "638b97f575fa4560b644e5fbfc9056be"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "7a5ffa399dbfe8bef76d1dcb2b9b8417"
server_key_iv: "824f739cf210605a8c51e6290bc5f270"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.574182 31376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "638b97f575fa4560b644e5fbfc9056be"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "7a5ffa399dbfe8bef76d1dcb2b9b8417"
server_key_iv: "824f739cf210605a8c51e6290bc5f270"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.577683 31376 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:02:07.579962 31391 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:07.580972 31376 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:07.581077 31376 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "638b97f575fa4560b644e5fbfc9056be"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "7a5ffa399dbfe8bef76d1dcb2b9b8417"
server_key_iv: "824f739cf210605a8c51e6290bc5f270"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.581178 31376 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:07.598513 31376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:07.601495 31376 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:07.601677 31376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:07.602237 31376 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:07.603137 31376 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:07.603185 31376 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:07.603226 31376 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:07.603258 31376 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:07.612304 31376 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:42647
I20251025 08:02:07.612342 31504 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:42647 every 8 connection(s)
I20251025 08:02:07.613231 31376 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:02:07.615569 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 31376
I20251025 08:02:07.615691 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:02:07.615916 28176 external_mini_cluster.cc:1468] Setting key 5075d013b795c294dd4737e101b1ae3d
Oct 25 08:02:07 dist-test-slave-lq8l krb5kdc[31128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
I20251025 08:02:07.626798 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:07.614919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:41375 (local address 127.27.132.62:44849)
1025 08:02:07.615091 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:07.615096 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:07.615790 (+ 694us) server_negotiation.cc:408] Connection header received
1025 08:02:07.616731 (+ 941us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:07.616734 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:07.616782 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:07.616871 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:07.618174 (+ 1303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.618961 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:07.619892 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.620023 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:07.623238 (+ 3215us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:07.623261 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:07.623264 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:07.623310 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:07.624901 (+ 1591us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:07.625437 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:07.625440 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:07.625442 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:07.625489 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:07.625731 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:07.625732 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:07.625734 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:07.625904 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:07.626001 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:07.626520 (+ 519us) server_negotiation.cc:300] Negotiation successful
1025 08:02:07.626655 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
I20251025 08:02:07.627490 31507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:07.615170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:44849 (local address 127.27.132.2:41375)
1025 08:02:07.615652 (+ 482us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:07.615697 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:07.616524 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:07.617004 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:07.617012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:07.617470 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:07.618017 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:07.618029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.619087 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:07.619092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:07.619777 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:07.619791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.620544 (+ 753us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:07.621214 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:07.621249 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:07.623070 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:07.625021 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:07.625027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:07.625039 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:07.625299 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:07.625567 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:07.625569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:07.625570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:07.625661 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:07.626007 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:07.626013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:07.626243 (+ 230us) client_negotiation.cc:769] Sending connection context
1025 08:02:07.626423 (+ 180us) client_negotiation.cc:240] Negotiation successful
1025 08:02:07.626680 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":154,"threads_started":1}
I20251025 08:02:07.629638 31505 heartbeater.cc:344] Connected to a master server at 127.27.132.62:44849
I20251025 08:02:07.629910 31505 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:07.630577 31505 heartbeater.cc:507] Master 127.27.132.62:44849 requested a full tablet report, sending...
I20251025 08:02:07.631716 31176 ts_manager.cc:194] Registered new tserver with Master: 638b97f575fa4560b644e5fbfc9056be (127.27.132.2:42647)
I20251025 08:02:07.632262 31176 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:41375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:07.663174 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:44849
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:07.771274 31512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:07.771595 31512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:07.771684 31512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:07.774907 31512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:07.774981 31512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:07.775096 31512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:02:07.779636 31512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:44849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:07.780725 31512 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:07.781594 31512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:07.788102 31517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:07.788234 31520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:07.788375 31512 server_base.cc:1047] running on GCE node
W20251025 08:02:07.788554 31518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:07.788987 31512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:07.789546 31512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:07.790701 31512 hybrid_clock.cc:648] HybridClock initialized: now 1761379327790693 us; error 31 us; skew 500 ppm
Oct 25 08:02:07 dist-test-slave-lq8l krb5kdc[31128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:07.793522 31512 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:02:07.794521 31512 webserver.cc:492] Webserver started at http://127.27.132.3:42449/ using document root <none> and password file <none>
I20251025 08:02:07.795050 31512 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:07.795096 31512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:07.795248 31512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:07.797030 31512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "828147f9707d47aea32071b77a1df9a6"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "c83c905d1626ace4cdfcb34db07015e8"
server_key_iv: "187f722bec1b7965b9305dca4b98e882"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.797461 31512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "828147f9707d47aea32071b77a1df9a6"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "c83c905d1626ace4cdfcb34db07015e8"
server_key_iv: "187f722bec1b7965b9305dca4b98e882"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.800916 31512 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:02:07.803151 31527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:07.804351 31512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:07.804459 31512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "828147f9707d47aea32071b77a1df9a6"
format_stamp: "Formatted at 2025-10-25 08:02:07 on dist-test-slave-lq8l"
server_key: "c83c905d1626ace4cdfcb34db07015e8"
server_key_iv: "187f722bec1b7965b9305dca4b98e882"
server_key_version: "encryptionkey@0"
I20251025 08:02:07.804581 31512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:07.830454 31512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:07.833673 31512 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:07.833861 31512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:07.834424 31512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:07.835315 31512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:07.835398 31512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:07.835453 31512 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:07.835489 31512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:07.844368 31512 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:38935
I20251025 08:02:07.844410 31640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:38935 every 8 connection(s)
I20251025 08:02:07.845412 31512 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:02:07.849531 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 31512
I20251025 08:02:07.849651 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:02:07.849895 28176 external_mini_cluster.cc:1468] Setting key e216ba773c0c86cee7d699679a5a3fc2
Oct 25 08:02:07 dist-test-slave-lq8l krb5kdc[31128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:07.858410 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:07.847223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:53653 (local address 127.27.132.62:44849)
1025 08:02:07.847403 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:07.847406 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:07.848116 (+ 710us) server_negotiation.cc:408] Connection header received
1025 08:02:07.848922 (+ 806us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:07.848925 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:07.848977 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:07.849054 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:07.850476 (+ 1422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.851178 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:07.852036 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.852209 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:07.854971 (+ 2762us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:07.854987 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:07.854989 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:07.855019 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:07.856589 (+ 1570us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:07.857072 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:07.857075 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:07.857076 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:07.857117 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:07.857429 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:07.857431 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:07.857432 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:07.857587 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:07.857670 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:07.858167 (+ 497us) server_negotiation.cc:300] Negotiation successful
1025 08:02:07.858275 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251025 08:02:07.859200 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:07.847534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:44849 (local address 127.27.132.3:53653)
1025 08:02:07.847966 (+ 432us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:07.848004 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:07.848729 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:07.849200 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:07.849207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:07.849614 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:07.850297 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:07.850319 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.851310 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:07.851315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:07.851918 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:07.851931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:07.852556 (+ 625us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:07.853162 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:07.853189 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:07.854791 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:07.856717 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:07.856722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:07.856732 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:07.856985 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:07.857202 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:07.857205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:07.857207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:07.857343 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:07.857668 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:07.857674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:07.857937 (+ 263us) client_negotiation.cc:769] Sending connection context
1025 08:02:07.858121 (+ 184us) client_negotiation.cc:240] Negotiation successful
1025 08:02:07.858421 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
I20251025 08:02:07.860430 31641 heartbeater.cc:344] Connected to a master server at 127.27.132.62:44849
I20251025 08:02:07.860713 31641 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:07.861271 31641 heartbeater.cc:507] Master 127.27.132.62:44849 requested a full tablet report, sending...
I20251025 08:02:07.862385 31176 ts_manager.cc:194] Registered new tserver with Master: 828147f9707d47aea32071b77a1df9a6 (127.27.132.3:38935)
I20251025 08:02:07.863119 31176 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:53653
I20251025 08:02:07.863868 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251025 08:02:07.999730 31215 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:07.996107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.27.132.62:44849)
1025 08:02:07.996249 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:07.996253 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:07.997323 (+ 1070us) server_negotiation.cc:408] Connection header received
1025 08:02:07.998226 (+ 903us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:07.998229 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:07.998279 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:07.998355 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:07.999586 (+ 1231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45902: BlockingRecv error: recv got EOF from 127.0.0.1:45902 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20251025 08:02:08.001159 31215 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:08.000324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.27.132.62:44849)
1025 08:02:08.000488 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:08.000492 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:08.000506 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:02:08.000580 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:08.000582 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:08.000620 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:08.000695 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:08.001073 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45914: BlockingRecv error: recv got EOF from 127.0.0.1:45914 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20251025 08:02:08.002209 31215 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:08.001384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.27.132.62:44849)
1025 08:02:08.001500 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:08.001504 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:08.001578 (+ 74us) server_negotiation.cc:408] Connection header received
1025 08:02:08.001693 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:08.001696 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:08.001744 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:08.001811 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:08.002135 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45920: BlockingRecv error: recv got EOF from 127.0.0.1:45920 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20251025 08:02:08.006261 31215 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:08.005492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.27.132.62:44849)
1025 08:02:08.005617 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:08.005620 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:08.005632 (+ 12us) server_negotiation.cc:408] Connection header received
1025 08:02:08.005667 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:08.005669 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:08.005716 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:08.005786 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:08.006183 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45930: BlockingRecv error: recv got EOF from 127.0.0.1:45930 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 08:02:08.064939 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 31240
I20251025 08:02:08.070755 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 31376
I20251025 08:02:08.076277 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 31512
I20251025 08:02:08.081918 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 31144
2025-10-25T08:02:08Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3277 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:02:08 dist-test-slave-lq8l krb5kdc[31664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:02:08 dist-test-slave-lq8l krb5kdc[31664](info): set up 2 sockets
Oct 25 08:02:08 dist-test-slave-lq8l krb5kdc[31664](info): commencing operation
krb5kdc: starting...
W20251025 08:02:10.076763 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.972s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:02:10 dist-test-slave-lq8l krb5kdc[31664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:02:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:10.222702 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:40391
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:38233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:40391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:10.329095 31680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:10.329345 31680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:10.329447 31680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:10.332723 31680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:10.332791 31680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:10.332827 31680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:10.332856 31680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:10.332890 31680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:10.337258 31680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:38233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:40391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:40391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:10.338459 31680 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:10.339347 31680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:10.345036 31685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:10.345193 31688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:10.345041 31686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:10.345335 31680 server_base.cc:1047] running on GCE node
I20251025 08:02:10.345836 31680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:10.346767 31680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:10.347927 31680 hybrid_clock.cc:648] HybridClock initialized: now 1761379330347916 us; error 29 us; skew 500 ppm
Oct 25 08:02:10 dist-test-slave-lq8l krb5kdc[31664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379330, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:10.350579 31680 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:02:10.351691 31680 webserver.cc:492] Webserver started at http://127.27.132.62:40287/ using document root <none> and password file <none>
I20251025 08:02:10.352217 31680 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:10.352263 31680 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:10.352427 31680 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:10.354130 31680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "e5e1dc4e95ee4d2a9c65e94f97a9fd1e"
format_stamp: "Formatted at 2025-10-25 08:02:10 on dist-test-slave-lq8l"
server_key: "a74956e525256c9a19d9b179201b9fdc"
server_key_iv: "9a1939372691cc77d12466e6b72b8221"
server_key_version: "encryptionkey@0"
I20251025 08:02:10.354579 31680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "e5e1dc4e95ee4d2a9c65e94f97a9fd1e"
format_stamp: "Formatted at 2025-10-25 08:02:10 on dist-test-slave-lq8l"
server_key: "a74956e525256c9a19d9b179201b9fdc"
server_key_iv: "9a1939372691cc77d12466e6b72b8221"
server_key_version: "encryptionkey@0"
I20251025 08:02:10.357999 31680 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251025 08:02:10.360216 31695 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:10.361228 31680 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:10.361346 31680 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "e5e1dc4e95ee4d2a9c65e94f97a9fd1e"
format_stamp: "Formatted at 2025-10-25 08:02:10 on dist-test-slave-lq8l"
server_key: "a74956e525256c9a19d9b179201b9fdc"
server_key_iv: "9a1939372691cc77d12466e6b72b8221"
server_key_version: "encryptionkey@0"
I20251025 08:02:10.361444 31680 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:10.378680 31680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:10.385470 31680 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:10.385682 31680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:10.393349 31747 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:40391 every 8 connection(s)
I20251025 08:02:10.393357 31680 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:40391
I20251025 08:02:10.394464 31680 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:10.397374 31748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:10.398746 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 31680
I20251025 08:02:10.398865 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:10.399099 28176 external_mini_cluster.cc:1468] Setting key 8d637ccf0f0f46b033f39b530a31b5f6
I20251025 08:02:10.404160 31748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e: Bootstrap starting.
I20251025 08:02:10.406603 31748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e: Neither blocks nor log segments found. Creating new log.
Oct 25 08:02:10 dist-test-slave-lq8l krb5kdc[31664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:10.407379 31748 log.cc:826] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:10.409715 31748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e: No bootstrap required, opened a new log
I20251025 08:02:10.412967 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:10.400445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.27.132.62:40391)
1025 08:02:10.400915 (+ 470us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:10.400929 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:10.400974 (+ 45us) server_negotiation.cc:408] Connection header received
1025 08:02:10.401660 (+ 686us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:10.401688 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:10.402066 (+ 378us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:10.402451 (+ 385us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:10.403397 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:10.404567 (+ 1170us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:10.405266 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:10.405575 (+ 309us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:10.407786 (+ 2211us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:10.407810 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:10.407826 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:10.407869 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:10.410140 (+ 2271us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:10.410616 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:10.410621 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:10.410627 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:10.410726 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:10.410943 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:10.410947 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:10.410949 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:10.411421 (+ 472us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:10.411636 (+ 215us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:10.411888 (+ 252us) server_negotiation.cc:300] Negotiation successful
1025 08:02:10.412159 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":130,"threads_started":1}
I20251025 08:02:10.413139 31748 raft_consensus.cc:359] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1dc4e95ee4d2a9c65e94f97a9fd1e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40391 } }
I20251025 08:02:10.413363 31748 raft_consensus.cc:385] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:10.413439 31748 raft_consensus.cc:740] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e1dc4e95ee4d2a9c65e94f97a9fd1e, State: Initialized, Role: FOLLOWER
I20251025 08:02:10.413913 31748 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1dc4e95ee4d2a9c65e94f97a9fd1e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40391 } }
I20251025 08:02:10.414062 31748 raft_consensus.cc:399] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:10.414124 31748 raft_consensus.cc:493] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:10.414217 31748 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:10.415226 31748 raft_consensus.cc:515] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1dc4e95ee4d2a9c65e94f97a9fd1e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40391 } }
I20251025 08:02:10.415638 31748 leader_election.cc:304] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5e1dc4e95ee4d2a9c65e94f97a9fd1e; no voters:
I20251025 08:02:10.415980 28176 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251025 08:02:10.415998 31748 leader_election.cc:290] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:10.416100 31753 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:10.416318 31753 raft_consensus.cc:697] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [term 1 LEADER]: Becoming Leader. State: Replica: e5e1dc4e95ee4d2a9c65e94f97a9fd1e, State: Running, Role: LEADER
I20251025 08:02:10.416702 31753 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1dc4e95ee4d2a9c65e94f97a9fd1e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40391 } }
I20251025 08:02:10.417294 31748 sys_catalog.cc:565] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:10.417814 31753 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5e1dc4e95ee4d2a9c65e94f97a9fd1e. Latest consensus state: current_term: 1 leader_uuid: "e5e1dc4e95ee4d2a9c65e94f97a9fd1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1dc4e95ee4d2a9c65e94f97a9fd1e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40391 } } }
I20251025 08:02:10.418043 31753 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:10.418183 31755 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5e1dc4e95ee4d2a9c65e94f97a9fd1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1dc4e95ee4d2a9c65e94f97a9fd1e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40391 } } }
I20251025 08:02:10.418277 31755 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:10.418576 31758 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:10.422210 31758 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:10.429659 31758 catalog_manager.cc:1357] Generated new cluster ID: 82595567c90d4c5ba1c65a645677de12
I20251025 08:02:10.429734 31758 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:10.435766 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:10.430119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.27.132.62:40391)
1025 08:02:10.430243 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:10.430248 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:10.430271 (+ 23us) server_negotiation.cc:408] Connection header received
1025 08:02:10.430470 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:10.430474 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:10.430530 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:10.430640 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:10.431621 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:10.432338 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:10.433275 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:10.433466 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:10.434135 (+ 669us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:10.434155 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:10.434159 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:10.435193 (+ 1034us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:10.435231 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:10.435250 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:10.435380 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:10.435536 (+ 156us) server_negotiation.cc:300] Negotiation successful
1025 08:02:10.435607 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 08:02:10.440647 31758 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:10.441963 31758 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:10.455056 31758 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e5e1dc4e95ee4d2a9c65e94f97a9fd1e: Generated new TSK 0
I20251025 08:02:10.456058 31758 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 08:02:10.467872 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 31680
2025-10-25T08:02:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2383 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:02:10 dist-test-slave-lq8l krb5kdc[31781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:02:10 dist-test-slave-lq8l krb5kdc[31781](info): set up 2 sockets
Oct 25 08:02:10 dist-test-slave-lq8l krb5kdc[31781](info): commencing operation
krb5kdc: starting...
W20251025 08:02:12.476521 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.988s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:02:12 dist-test-slave-lq8l krb5kdc[31781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:02:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:12.619091 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:35779
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:35779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:12.727756 31797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:12.728008 31797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:12.728067 31797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:12.731446 31797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:12.731516 31797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:12.731540 31797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:12.731559 31797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:12.731576 31797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:12.736035 31797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:35779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:35779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:12.737195 31797 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:12.738102 31797 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:12.743947 31802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:12.744077 31797 server_base.cc:1047] running on GCE node
W20251025 08:02:12.743947 31805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:12.743952 31803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:12.744755 31797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:12.745677 31797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:12.746842 31797 hybrid_clock.cc:648] HybridClock initialized: now 1761379332746830 us; error 36 us; skew 500 ppm
Oct 25 08:02:12 dist-test-slave-lq8l krb5kdc[31781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:12.749900 31797 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:02:12.751015 31797 webserver.cc:492] Webserver started at http://127.27.132.62:37927/ using document root <none> and password file <none>
I20251025 08:02:12.751660 31797 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:12.751715 31797 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:12.751878 31797 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:12.753583 31797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "17631c3be2ac4fff92ad46a28fb9dc2f"
format_stamp: "Formatted at 2025-10-25 08:02:12 on dist-test-slave-lq8l"
server_key: "a7029c1dbccaf45242a0d57ae6d4efe7"
server_key_iv: "7ebd1d1f865a0f114a4cb0854d30b9de"
server_key_version: "encryptionkey@0"
I20251025 08:02:12.754030 31797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "17631c3be2ac4fff92ad46a28fb9dc2f"
format_stamp: "Formatted at 2025-10-25 08:02:12 on dist-test-slave-lq8l"
server_key: "a7029c1dbccaf45242a0d57ae6d4efe7"
server_key_iv: "7ebd1d1f865a0f114a4cb0854d30b9de"
server_key_version: "encryptionkey@0"
I20251025 08:02:12.757591 31797 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:02:12.759912 31812 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:12.760988 31797 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:12.761093 31797 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "17631c3be2ac4fff92ad46a28fb9dc2f"
format_stamp: "Formatted at 2025-10-25 08:02:12 on dist-test-slave-lq8l"
server_key: "a7029c1dbccaf45242a0d57ae6d4efe7"
server_key_iv: "7ebd1d1f865a0f114a4cb0854d30b9de"
server_key_version: "encryptionkey@0"
I20251025 08:02:12.761193 31797 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:12.777878 31797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:12.781045 31797 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:12.781203 31797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:12.788407 31797 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:35779
I20251025 08:02:12.788419 31864 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:35779 every 8 connection(s)
I20251025 08:02:12.789497 31797 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:12.792546 31865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:12.795452 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 31797
I20251025 08:02:12.795593 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:12.795838 28176 external_mini_cluster.cc:1468] Setting key 8d28b63796e0de78688aff50ccfec5cd
I20251025 08:02:12.799407 31865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f: Bootstrap starting.
I20251025 08:02:12.802079 31865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:12.803278 31865 log.cc:826] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f: Log is configured to *not* fsync() on all Append() calls
Oct 25 08:02:12 dist-test-slave-lq8l krb5kdc[31781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:12.805590 31865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f: No bootstrap required, opened a new log
I20251025 08:02:12.809178 31865 raft_consensus.cc:359] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17631c3be2ac4fff92ad46a28fb9dc2f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35779 } }
I20251025 08:02:12.809414 31865 raft_consensus.cc:385] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:12.809490 31865 raft_consensus.cc:740] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17631c3be2ac4fff92ad46a28fb9dc2f, State: Initialized, Role: FOLLOWER
I20251025 08:02:12.809979 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:12.797142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.27.132.62:35779)
1025 08:02:12.797617 (+ 475us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:12.797630 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:12.797681 (+ 51us) server_negotiation.cc:408] Connection header received
1025 08:02:12.798425 (+ 744us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:12.798454 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:12.798850 (+ 396us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:12.799239 (+ 389us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:12.800200 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:12.801401 (+ 1201us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:12.802099 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:12.802404 (+ 305us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:12.804806 (+ 2402us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:12.804830 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:12.804845 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:12.804888 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:12.807068 (+ 2180us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:12.807552 (+ 484us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:12.807558 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:12.807565 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:12.807665 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:12.807899 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:12.807902 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:12.807904 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:12.808371 (+ 467us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:12.808649 (+ 278us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:12.808876 (+ 227us) server_negotiation.cc:300] Negotiation successful
1025 08:02:12.809147 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":133,"threads_started":1}
I20251025 08:02:12.809975 31865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17631c3be2ac4fff92ad46a28fb9dc2f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35779 } }
I20251025 08:02:12.810256 31865 raft_consensus.cc:399] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:12.810334 31865 raft_consensus.cc:493] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:12.810623 31865 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:12.811630 31865 raft_consensus.cc:515] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17631c3be2ac4fff92ad46a28fb9dc2f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35779 } }
I20251025 08:02:12.811982 31865 leader_election.cc:304] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17631c3be2ac4fff92ad46a28fb9dc2f; no voters:
I20251025 08:02:12.812373 31865 leader_election.cc:290] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:12.812470 31870 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:12.812695 31870 raft_consensus.cc:697] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [term 1 LEADER]: Becoming Leader. State: Replica: 17631c3be2ac4fff92ad46a28fb9dc2f, State: Running, Role: LEADER
I20251025 08:02:12.813043 31870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17631c3be2ac4fff92ad46a28fb9dc2f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35779 } }
I20251025 08:02:12.813530 31865 sys_catalog.cc:565] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:12.815064 31872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17631c3be2ac4fff92ad46a28fb9dc2f. Latest consensus state: current_term: 1 leader_uuid: "17631c3be2ac4fff92ad46a28fb9dc2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17631c3be2ac4fff92ad46a28fb9dc2f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35779 } } }
I20251025 08:02:12.815078 31871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17631c3be2ac4fff92ad46a28fb9dc2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17631c3be2ac4fff92ad46a28fb9dc2f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35779 } } }
I20251025 08:02:12.815243 31871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:12.815951 31872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:12.816247 31879 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:12.818989 31879 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:12.825217 31879 catalog_manager.cc:1357] Generated new cluster ID: 670e4be4043f409d98b47a1cc1b265b8
I20251025 08:02:12.825295 31879 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:12.846992 31879 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:12.848019 31879 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:12.853865 31879 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 17631c3be2ac4fff92ad46a28fb9dc2f: Generated new TSK 0
I20251025 08:02:12.854604 31879 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:12.902714 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:35779
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:13.018999 31893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:13.019265 31893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:13.019346 31893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:13.022702 31893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:13.022778 31893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:13.022861 31893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:02:13.027256 31893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:35779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:13.028466 31893 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:13.029314 31893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:13.036247 31898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:13.036494 31893 server_base.cc:1047] running on GCE node
W20251025 08:02:13.036302 31899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:13.036247 31901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:13.037050 31893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:13.037662 31893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:13.038847 31893 hybrid_clock.cc:648] HybridClock initialized: now 1761379333038835 us; error 37 us; skew 500 ppm
Oct 25 08:02:13 dist-test-slave-lq8l krb5kdc[31781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:13.041879 31893 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:02:13.043015 31893 webserver.cc:492] Webserver started at http://127.27.132.1:44627/ using document root <none> and password file <none>
I20251025 08:02:13.043666 31893 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:13.043730 31893 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:13.043933 31893 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:13.046095 31893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "1e37ddd76fa9446187e5de4a602f8245"
format_stamp: "Formatted at 2025-10-25 08:02:13 on dist-test-slave-lq8l"
server_key: "a44f0e7ff46276f8bc017fabd970ea16"
server_key_iv: "a14f559cf1e52ff0ae9099fab85bad05"
server_key_version: "encryptionkey@0"
I20251025 08:02:13.046658 31893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "1e37ddd76fa9446187e5de4a602f8245"
format_stamp: "Formatted at 2025-10-25 08:02:13 on dist-test-slave-lq8l"
server_key: "a44f0e7ff46276f8bc017fabd970ea16"
server_key_iv: "a14f559cf1e52ff0ae9099fab85bad05"
server_key_version: "encryptionkey@0"
I20251025 08:02:13.050405 31893 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251025 08:02:13.052997 31908 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:13.054231 31893 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:13.054373 31893 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "1e37ddd76fa9446187e5de4a602f8245"
format_stamp: "Formatted at 2025-10-25 08:02:13 on dist-test-slave-lq8l"
server_key: "a44f0e7ff46276f8bc017fabd970ea16"
server_key_iv: "a14f559cf1e52ff0ae9099fab85bad05"
server_key_version: "encryptionkey@0"
I20251025 08:02:13.054499 31893 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:13.079278 31893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:13.082408 31893 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:13.082624 31893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:13.083227 31893 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:13.084172 31893 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:13.084225 31893 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:13.084286 31893 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:13.084318 31893 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:13.094318 31893 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:43407
I20251025 08:02:13.094372 32021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:43407 every 8 connection(s)
I20251025 08:02:13.095420 31893 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:02:13.099714 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 31893
I20251025 08:02:13.099824 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:02:13.100052 28176 external_mini_cluster.cc:1468] Setting key 8e652455de485cd2962b5581f35ac03c
Oct 25 08:02:13 dist-test-slave-lq8l krb5kdc[31781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
I20251025 08:02:13.109385 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:13.097401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:33347 (local address 127.27.132.62:35779)
1025 08:02:13.097549 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:13.097553 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:13.098286 (+ 733us) server_negotiation.cc:408] Connection header received
1025 08:02:13.099176 (+ 890us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:13.099180 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:13.099248 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:13.099362 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:13.100942 (+ 1580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.101553 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:13.102266 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.102500 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:13.105324 (+ 2824us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:13.105348 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:13.105350 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:13.105379 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:13.107060 (+ 1681us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:13.107686 (+ 626us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:13.107690 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:13.107691 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:13.107752 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:13.108072 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:13.108075 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:13.108076 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:13.108249 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:13.108344 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:13.109054 (+ 710us) server_negotiation.cc:300] Negotiation successful
1025 08:02:13.109210 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251025 08:02:13.110105 32024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:13.097658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:35779 (local address 127.27.132.1:33347)
1025 08:02:13.098126 (+ 468us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:13.098165 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:13.098949 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:13.099562 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:13.099571 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:13.100060 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:13.100770 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:13.100788 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.101718 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:13.101721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:13.102126 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:13.102134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.102329 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:13.103561 (+ 1232us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:13.103589 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:13.105136 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:13.107212 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:13.107219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:13.107235 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:13.107569 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:13.107857 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:13.107859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:13.107861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:13.107961 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:13.108371 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:13.108378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:13.108657 (+ 279us) client_negotiation.cc:769] Sending connection context
1025 08:02:13.108861 (+ 204us) client_negotiation.cc:240] Negotiation successful
1025 08:02:13.109265 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
I20251025 08:02:13.111500 32022 heartbeater.cc:344] Connected to a master server at 127.27.132.62:35779
I20251025 08:02:13.111757 32022 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
I20251025 08:02:13.112514 32022 heartbeater.cc:507] Master 127.27.132.62:35779 requested a full tablet report, sending...
I20251025 08:02:13.114356 31829 ts_manager.cc:194] Registered new tserver with Master: 1e37ddd76fa9446187e5de4a602f8245 (127.27.132.1:43407)
I20251025 08:02:13.115708 31829 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:33347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:13.158917 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:35779
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:13.269817 32029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:13.270100 32029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:13.270198 32029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:13.273972 32029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:13.274070 32029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:13.274222 32029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:02:13.278797 32029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:35779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:13.280037 32029 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:13.280942 32029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:13.288172 32037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:13.288277 32029 server_base.cc:1047] running on GCE node
W20251025 08:02:13.288146 32035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:13.288146 32034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:13.288820 32029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:13.289440 32029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:13.290589 32029 hybrid_clock.cc:648] HybridClock initialized: now 1761379333290576 us; error 36 us; skew 500 ppm
Oct 25 08:02:13 dist-test-slave-lq8l krb5kdc[31781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:13.293725 32029 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:02:13.294960 32029 webserver.cc:492] Webserver started at http://127.27.132.2:44761/ using document root <none> and password file <none>
I20251025 08:02:13.295617 32029 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:13.295676 32029 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:13.295886 32029 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:13.297722 32029 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "55e672432ee446088924647376e7eab2"
format_stamp: "Formatted at 2025-10-25 08:02:13 on dist-test-slave-lq8l"
server_key: "132345b7898ef08eb781793c8fa2fd76"
server_key_iv: "06604767b540fd70414e2f4b3e6959b5"
server_key_version: "encryptionkey@0"
I20251025 08:02:13.298231 32029 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "55e672432ee446088924647376e7eab2"
format_stamp: "Formatted at 2025-10-25 08:02:13 on dist-test-slave-lq8l"
server_key: "132345b7898ef08eb781793c8fa2fd76"
server_key_iv: "06604767b540fd70414e2f4b3e6959b5"
server_key_version: "encryptionkey@0"
I20251025 08:02:13.302030 32029 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251025 08:02:13.304612 32044 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:13.305737 32029 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:13.305855 32029 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "55e672432ee446088924647376e7eab2"
format_stamp: "Formatted at 2025-10-25 08:02:13 on dist-test-slave-lq8l"
server_key: "132345b7898ef08eb781793c8fa2fd76"
server_key_iv: "06604767b540fd70414e2f4b3e6959b5"
server_key_version: "encryptionkey@0"
I20251025 08:02:13.305959 32029 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:13.321300 32029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:13.324537 32029 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:13.324750 32029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:13.325307 32029 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:13.326236 32029 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:13.326287 32029 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:13.326349 32029 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:13.326382 32029 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:13.335562 32029 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:43815
I20251025 08:02:13.335606 32157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:43815 every 8 connection(s)
I20251025 08:02:13.336553 32029 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
Oct 25 08:02:13 dist-test-slave-lq8l krb5kdc[31781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:13.345392 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 32029
I20251025 08:02:13.345506 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:02:13.345768 28176 external_mini_cluster.cc:1468] Setting key 39096f9da3a4daa49dab5316a588d75c
I20251025 08:02:13.350195 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:13.338325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:53557 (local address 127.27.132.62:35779)
1025 08:02:13.338471 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:13.338475 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:13.339167 (+ 692us) server_negotiation.cc:408] Connection header received
1025 08:02:13.340070 (+ 903us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:13.340074 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:13.340139 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:13.340215 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:13.341601 (+ 1386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.342105 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:13.342775 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.342968 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:13.345976 (+ 3008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:13.346003 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:13.346007 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:13.346043 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:13.347818 (+ 1775us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:13.348522 (+ 704us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:13.348525 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:13.348526 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:13.348569 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:13.348968 (+ 399us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:13.348975 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:13.348976 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:13.349201 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:13.349344 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:13.349937 (+ 593us) server_negotiation.cc:300] Negotiation successful
1025 08:02:13.350068 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251025 08:02:13.350864 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:13.338548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:35779 (local address 127.27.132.2:53557)
1025 08:02:13.339005 (+ 457us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:13.339044 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:13.339845 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:13.340405 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:13.340414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:13.340862 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:13.341431 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:13.341445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.342237 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:13.342240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:13.342647 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:13.342654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.342885 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:13.344168 (+ 1283us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:13.344192 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:13.345775 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:13.347965 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:13.347975 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:13.347991 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:13.348382 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:13.348657 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:13.348663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:13.348668 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:13.348816 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:13.349350 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:13.349359 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:13.349644 (+ 285us) client_negotiation.cc:769] Sending connection context
1025 08:02:13.349843 (+ 199us) client_negotiation.cc:240] Negotiation successful
1025 08:02:13.350085 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
I20251025 08:02:13.352296 32158 heartbeater.cc:344] Connected to a master server at 127.27.132.62:35779
I20251025 08:02:13.352563 32158 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:13.353170 32158 heartbeater.cc:507] Master 127.27.132.62:35779 requested a full tablet report, sending...
I20251025 08:02:13.354470 31829 ts_manager.cc:194] Registered new tserver with Master: 55e672432ee446088924647376e7eab2 (127.27.132.2:43815)
I20251025 08:02:13.355370 31829 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:53557
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:13.398860 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:35779
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:13.511739 32165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:13.512012 32165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:13.512072 32165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:13.515580 32165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:13.515661 32165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:13.515746 32165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:02:13.520351 32165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:35779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:13.521497 32165 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:13.522370 32165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:13.529618 32170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:13.529618 32171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:13.529724 32173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:13.529875 32165 server_base.cc:1047] running on GCE node
I20251025 08:02:13.530439 32165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:13.531090 32165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:13.532250 32165 hybrid_clock.cc:648] HybridClock initialized: now 1761379333532227 us; error 46 us; skew 500 ppm
Oct 25 08:02:13 dist-test-slave-lq8l krb5kdc[31781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:13.539464 32165 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:02:13.540671 32165 webserver.cc:492] Webserver started at http://127.27.132.3:37263/ using document root <none> and password file <none>
I20251025 08:02:13.541250 32165 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:13.541301 32165 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:13.541467 32165 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:13.543202 32165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "39d45353969f4605b578b7434089184c"
format_stamp: "Formatted at 2025-10-25 08:02:13 on dist-test-slave-lq8l"
server_key: "e1080b0d2a098608fec5bec00ea9db0d"
server_key_iv: "cb8f147eff2e8de0b3b5f0d034145649"
server_key_version: "encryptionkey@0"
I20251025 08:02:13.543805 32165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "39d45353969f4605b578b7434089184c"
format_stamp: "Formatted at 2025-10-25 08:02:13 on dist-test-slave-lq8l"
server_key: "e1080b0d2a098608fec5bec00ea9db0d"
server_key_iv: "cb8f147eff2e8de0b3b5f0d034145649"
server_key_version: "encryptionkey@0"
I20251025 08:02:13.547613 32165 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:02:13.550082 32180 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:13.551369 32165 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:13.551493 32165 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "39d45353969f4605b578b7434089184c"
format_stamp: "Formatted at 2025-10-25 08:02:13 on dist-test-slave-lq8l"
server_key: "e1080b0d2a098608fec5bec00ea9db0d"
server_key_iv: "cb8f147eff2e8de0b3b5f0d034145649"
server_key_version: "encryptionkey@0"
I20251025 08:02:13.551591 32165 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:13.562386 32165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:13.565567 32165 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:13.565742 32165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:13.566247 32165 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:13.567118 32165 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:13.567168 32165 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:13.567209 32165 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:13.567231 32165 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:13.576525 32165 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:43799
I20251025 08:02:13.576563 32293 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:43799 every 8 connection(s)
I20251025 08:02:13.577520 32165 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:02:13.585557 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 32165
I20251025 08:02:13.585675 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:02:13.585912 28176 external_mini_cluster.cc:1468] Setting key cb2221270023ac22d4ef94ea2483f127
Oct 25 08:02:13 dist-test-slave-lq8l krb5kdc[31781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:13.591044 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:13.579498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:37479 (local address 127.27.132.62:35779)
1025 08:02:13.579667 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:13.579671 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:13.580411 (+ 740us) server_negotiation.cc:408] Connection header received
1025 08:02:13.581309 (+ 898us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:13.581313 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:13.581381 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:13.581461 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:13.582866 (+ 1405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.583437 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:13.584096 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.584278 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:13.587306 (+ 3028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:13.587343 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:13.587347 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:13.587387 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:13.588977 (+ 1590us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:13.589513 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:13.589516 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:13.589517 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:13.589566 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:13.589865 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:13.589868 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:13.589870 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:13.590036 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:13.590127 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:13.590726 (+ 599us) server_negotiation.cc:300] Negotiation successful
1025 08:02:13.590864 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251025 08:02:13.591768 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:13.579769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:35779 (local address 127.27.132.3:37479)
1025 08:02:13.580234 (+ 465us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:13.580273 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:13.581085 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:13.581660 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:13.581669 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:13.582097 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:13.582688 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:13.582701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.583565 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:13.583568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:13.583962 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:13.583969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.584201 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:13.585411 (+ 1210us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:13.585437 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:13.587129 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:13.589117 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:13.589123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:13.589137 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:13.589395 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:13.589663 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:13.589666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:13.589667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:13.589768 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:13.590150 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:13.590158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:13.590458 (+ 300us) client_negotiation.cc:769] Sending connection context
1025 08:02:13.590671 (+ 213us) client_negotiation.cc:240] Negotiation successful
1025 08:02:13.590978 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":101,"threads_started":1}
I20251025 08:02:13.592960 32294 heartbeater.cc:344] Connected to a master server at 127.27.132.62:35779
I20251025 08:02:13.593236 32294 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:13.593803 32294 heartbeater.cc:507] Master 127.27.132.62:35779 requested a full tablet report, sending...
I20251025 08:02:13.594887 31829 ts_manager.cc:194] Registered new tserver with Master: 39d45353969f4605b578b7434089184c (127.27.132.3:43799)
I20251025 08:02:13.595484 31829 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:37479
I20251025 08:02:13.600265 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 25 08:02:13 dist-test-slave-lq8l krb5kdc[31781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379333, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Oct 25 08:02:13 dist-test-slave-lq8l krb5kdc[31781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379333, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:13.624117 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:13.616438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.27.132.62:35779)
1025 08:02:13.616643 (+ 205us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:13.616647 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:13.616741 (+ 94us) server_negotiation.cc:408] Connection header received
1025 08:02:13.616847 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:13.616850 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:13.616903 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:13.616979 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:13.617793 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.618284 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:13.618902 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:13.619021 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:13.621245 (+ 2224us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:13.621267 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:13.621269 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:13.621296 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:13.622592 (+ 1296us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:13.622961 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:13.622964 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:13.622965 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:13.623002 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:13.623267 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:13.623269 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:13.623270 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:13.623506 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:13.623617 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:13.623827 (+ 210us) server_negotiation.cc:300] Negotiation successful
1025 08:02:13.623930 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20251025 08:02:13.624568 31829 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36684
I20251025 08:02:13.625623 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 31893
I20251025 08:02:13.631419 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 32029
I20251025 08:02:13.637209 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 32165
I20251025 08:02:13.642470 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 31797
2025-10-25T08:02:13Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3176 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:02:13 dist-test-slave-lq8l krb5kdc[32306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:02:13 dist-test-slave-lq8l krb5kdc[32306](info): set up 2 sockets
Oct 25 08:02:13 dist-test-slave-lq8l krb5kdc[32306](info): commencing operation
krb5kdc: starting...
W20251025 08:02:15.636610 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.972s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:02:15 dist-test-slave-lq8l krb5kdc[32306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:02:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:15.783272 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:39153
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:39153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:15.893023 32322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:15.893306 32322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:15.893404 32322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:15.896816 32322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:15.896903 32322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:15.896936 32322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:15.896971 32322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:15.897001 32322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:15.901516 32322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:39153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:39153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:15.902765 32322 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:15.903738 32322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:15.909466 32327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:15.909502 32330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:15.909474 32328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:15.910055 32322 server_base.cc:1047] running on GCE node
I20251025 08:02:15.910622 32322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:15.911633 32322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:15.912817 32322 hybrid_clock.cc:648] HybridClock initialized: now 1761379335912802 us; error 38 us; skew 500 ppm
Oct 25 08:02:15 dist-test-slave-lq8l krb5kdc[32306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379335, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:15.915516 32322 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:02:15.916605 32322 webserver.cc:492] Webserver started at http://127.27.132.62:39911/ using document root <none> and password file <none>
I20251025 08:02:15.917153 32322 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:15.917204 32322 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:15.917398 32322 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:15.919126 32322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "5053da9330eb49a6acf8215c85b7422f"
format_stamp: "Formatted at 2025-10-25 08:02:15 on dist-test-slave-lq8l"
server_key: "865825a9c61556f8afc5696672939faa"
server_key_iv: "8eca1f91a3e117cb0a1c5ed09679309e"
server_key_version: "encryptionkey@0"
I20251025 08:02:15.919639 32322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "5053da9330eb49a6acf8215c85b7422f"
format_stamp: "Formatted at 2025-10-25 08:02:15 on dist-test-slave-lq8l"
server_key: "865825a9c61556f8afc5696672939faa"
server_key_iv: "8eca1f91a3e117cb0a1c5ed09679309e"
server_key_version: "encryptionkey@0"
I20251025 08:02:15.923111 32322 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251025 08:02:15.925604 32337 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:15.926756 32322 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:15.926872 32322 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "5053da9330eb49a6acf8215c85b7422f"
format_stamp: "Formatted at 2025-10-25 08:02:15 on dist-test-slave-lq8l"
server_key: "865825a9c61556f8afc5696672939faa"
server_key_iv: "8eca1f91a3e117cb0a1c5ed09679309e"
server_key_version: "encryptionkey@0"
I20251025 08:02:15.926975 32322 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:15.937150 32322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:15.945470 32322 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:15.945673 32322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:15.954412 32322 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:39153
I20251025 08:02:15.954423 32389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:39153 every 8 connection(s)
I20251025 08:02:15.955627 32322 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:15.958631 32390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:15.959901 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 32322
I20251025 08:02:15.960021 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:15.960290 28176 external_mini_cluster.cc:1468] Setting key ac720f83ec3f7cd285ef434c58b9b580
I20251025 08:02:15.965102 32390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f: Bootstrap starting.
I20251025 08:02:15.967707 32390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:15.968513 32390 log.cc:826] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f: Log is configured to *not* fsync() on all Append() calls
Oct 25 08:02:15 dist-test-slave-lq8l krb5kdc[32306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:15.970702 32390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f: No bootstrap required, opened a new log
I20251025 08:02:15.973241 32390 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5053da9330eb49a6acf8215c85b7422f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 39153 } }
I20251025 08:02:15.973474 32390 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:15.973552 32390 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5053da9330eb49a6acf8215c85b7422f, State: Initialized, Role: FOLLOWER
I20251025 08:02:15.973973 32390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5053da9330eb49a6acf8215c85b7422f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 39153 } }
I20251025 08:02:15.974100 32390 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:15.974169 32390 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:15.974238 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:15.961983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40966 (local address 127.27.132.62:39153)
1025 08:02:15.962483 (+ 500us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:15.962499 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:15.962616 (+ 117us) server_negotiation.cc:408] Connection header received
1025 08:02:15.963225 (+ 609us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:15.963244 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:15.963600 (+ 356us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:15.963916 (+ 316us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:15.964885 (+ 969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:15.965828 (+ 943us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:15.966546 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:15.966848 (+ 302us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:15.969485 (+ 2637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:15.969505 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:15.969517 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:15.969553 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:15.971635 (+ 2082us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:15.972077 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:15.972081 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:15.972086 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:15.972158 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:15.972400 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:15.972403 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:15.972405 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:15.972743 (+ 338us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:15.972948 (+ 205us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:15.973237 (+ 289us) server_negotiation.cc:300] Negotiation successful
1025 08:02:15.973517 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":132,"threads_started":1}
I20251025 08:02:15.974262 32390 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:15.975469 32390 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5053da9330eb49a6acf8215c85b7422f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 39153 } }
I20251025 08:02:15.975811 32390 leader_election.cc:304] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5053da9330eb49a6acf8215c85b7422f; no voters:
I20251025 08:02:15.976097 32390 leader_election.cc:290] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:15.976193 32395 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:15.976372 32395 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [term 1 LEADER]: Becoming Leader. State: Replica: 5053da9330eb49a6acf8215c85b7422f, State: Running, Role: LEADER
I20251025 08:02:15.976727 32395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5053da9330eb49a6acf8215c85b7422f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 39153 } }
I20251025 08:02:15.977289 32390 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:15.978492 32397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5053da9330eb49a6acf8215c85b7422f. Latest consensus state: current_term: 1 leader_uuid: "5053da9330eb49a6acf8215c85b7422f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5053da9330eb49a6acf8215c85b7422f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 39153 } } }
I20251025 08:02:15.978622 32397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:15.978955 32396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5053da9330eb49a6acf8215c85b7422f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5053da9330eb49a6acf8215c85b7422f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 39153 } } }
I20251025 08:02:15.979014 32404 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:15.979691 32396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:15.982923 32404 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:15.989254 32404 catalog_manager.cc:1357] Generated new cluster ID: 9203305e89ec4d65866eab427181b50d
I20251025 08:02:15.989346 32404 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:16.007783 32404 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:16.009171 32404 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:16.016901 32404 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5053da9330eb49a6acf8215c85b7422f: Generated new TSK 0
I20251025 08:02:16.017800 32404 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:16.066325 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:39153
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:16.178053 32418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:16.178289 32418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:16.178352 32418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:16.181926 32418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:16.181998 32418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:16.182101 32418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:02:16.186569 32418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:39153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:16.187783 32418 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:16.188696 32418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:16.195626 32423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:16.195782 32426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:16.195834 32418 server_base.cc:1047] running on GCE node
W20251025 08:02:16.195652 32424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:16.196292 32418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:16.197048 32418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:16.198252 32418 hybrid_clock.cc:648] HybridClock initialized: now 1761379336198232 us; error 41 us; skew 500 ppm
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[32306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:16.201148 32418 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:02:16.202289 32418 webserver.cc:492] Webserver started at http://127.27.132.1:45499/ using document root <none> and password file <none>
I20251025 08:02:16.202865 32418 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:16.202914 32418 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:16.203078 32418 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:16.204897 32418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "980decfa465643689e6365c2e739fa3d"
format_stamp: "Formatted at 2025-10-25 08:02:16 on dist-test-slave-lq8l"
server_key: "04882d17464dede699f0973ce8a19353"
server_key_iv: "31b388f7f43c3a8004d78085e44e5f7c"
server_key_version: "encryptionkey@0"
I20251025 08:02:16.205360 32418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "980decfa465643689e6365c2e739fa3d"
format_stamp: "Formatted at 2025-10-25 08:02:16 on dist-test-slave-lq8l"
server_key: "04882d17464dede699f0973ce8a19353"
server_key_iv: "31b388f7f43c3a8004d78085e44e5f7c"
server_key_version: "encryptionkey@0"
I20251025 08:02:16.209172 32418 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251025 08:02:16.212724 32433 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:16.214049 32418 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:02:16.214181 32418 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "980decfa465643689e6365c2e739fa3d"
format_stamp: "Formatted at 2025-10-25 08:02:16 on dist-test-slave-lq8l"
server_key: "04882d17464dede699f0973ce8a19353"
server_key_iv: "31b388f7f43c3a8004d78085e44e5f7c"
server_key_version: "encryptionkey@0"
I20251025 08:02:16.214301 32418 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:16.226099 32418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:16.229128 32418 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:16.229300 32418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:16.229813 32418 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:16.230690 32418 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:16.230737 32418 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:16.230777 32418 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:16.230798 32418 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:16.239799 32418 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:36537
I20251025 08:02:16.239846 32546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:36537 every 8 connection(s)
I20251025 08:02:16.240777 32418 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:02:16.242867 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 32418
I20251025 08:02:16.243005 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:02:16.243294 28176 external_mini_cluster.cc:1468] Setting key 2ea2073d6c67c7ccb3dabd16c28bb979
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[32306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
I20251025 08:02:16.253540 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:16.242496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:46477 (local address 127.27.132.62:39153)
1025 08:02:16.242638 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:16.242642 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:16.243498 (+ 856us) server_negotiation.cc:408] Connection header received
1025 08:02:16.244417 (+ 919us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:16.244421 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:16.244479 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:16.244563 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:16.246280 (+ 1717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.246792 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:16.247503 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.247700 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:16.249910 (+ 2210us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:16.249931 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:16.249934 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:16.249959 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:16.251566 (+ 1607us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:16.252055 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:16.252059 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:16.252061 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:16.252108 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:16.252386 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:16.252389 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:16.252391 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:16.252601 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:16.252714 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:16.253219 (+ 505us) server_negotiation.cc:300] Negotiation successful
1025 08:02:16.253366 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
I20251025 08:02:16.254384 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:16.242761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:39153 (local address 127.27.132.1:46477)
1025 08:02:16.243284 (+ 523us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:16.243320 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:16.244213 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:16.244747 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:16.244756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:16.245158 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:16.246110 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:16.246124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.246926 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:16.246929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:16.247380 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:16.247391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.247625 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:16.248271 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:16.248297 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:16.249751 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:16.251691 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:16.251697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:16.251709 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:16.251947 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:16.252198 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:16.252201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:16.252203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:16.252290 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:16.252713 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:16.252722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:16.252968 (+ 246us) client_negotiation.cc:769] Sending connection context
1025 08:02:16.253163 (+ 195us) client_negotiation.cc:240] Negotiation successful
1025 08:02:16.253529 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":156,"threads_started":1}
I20251025 08:02:16.255656 32547 heartbeater.cc:344] Connected to a master server at 127.27.132.62:39153
I20251025 08:02:16.255949 32547 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:16.256481 32547 heartbeater.cc:507] Master 127.27.132.62:39153 requested a full tablet report, sending...
I20251025 08:02:16.258081 32354 ts_manager.cc:194] Registered new tserver with Master: 980decfa465643689e6365c2e739fa3d (127.27.132.1:36537)
I20251025 08:02:16.259236 32354 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:46477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:16.292150 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:39153
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:16.400020 32554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:16.400269 32554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:16.400367 32554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:16.403661 32554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:16.403734 32554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:16.403847 32554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:02:16.408099 32554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:39153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:16.409197 32554 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:16.410070 32554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:16.416769 32559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:16.416736 32560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:16.416736 32562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:16.416818 32554 server_base.cc:1047] running on GCE node
I20251025 08:02:16.417376 32554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:16.417922 32554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:16.419080 32554 hybrid_clock.cc:648] HybridClock initialized: now 1761379336419066 us; error 28 us; skew 500 ppm
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[32306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:16.421751 32554 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:02:16.422765 32554 webserver.cc:492] Webserver started at http://127.27.132.2:41291/ using document root <none> and password file <none>
I20251025 08:02:16.423298 32554 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:16.423393 32554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:16.423578 32554 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:16.425339 32554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "90c635c3b56942c0a34728181fa1b737"
format_stamp: "Formatted at 2025-10-25 08:02:16 on dist-test-slave-lq8l"
server_key: "5906b7b3c91aa4d1fe26a6b690a733a1"
server_key_iv: "3523fb2b9a7d5b20d78281a8d7359fe1"
server_key_version: "encryptionkey@0"
I20251025 08:02:16.425783 32554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "90c635c3b56942c0a34728181fa1b737"
format_stamp: "Formatted at 2025-10-25 08:02:16 on dist-test-slave-lq8l"
server_key: "5906b7b3c91aa4d1fe26a6b690a733a1"
server_key_iv: "3523fb2b9a7d5b20d78281a8d7359fe1"
server_key_version: "encryptionkey@0"
I20251025 08:02:16.429203 32554 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:02:16.431480 32569 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:16.432488 32554 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:16.432585 32554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "90c635c3b56942c0a34728181fa1b737"
format_stamp: "Formatted at 2025-10-25 08:02:16 on dist-test-slave-lq8l"
server_key: "5906b7b3c91aa4d1fe26a6b690a733a1"
server_key_iv: "3523fb2b9a7d5b20d78281a8d7359fe1"
server_key_version: "encryptionkey@0"
I20251025 08:02:16.432665 32554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:16.443715 32554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:16.446692 32554 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:16.446848 32554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:16.447350 32554 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:16.448303 32554 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:16.448350 32554 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:16.448390 32554 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:16.448419 32554 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:16.457280 32554 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:46517
I20251025 08:02:16.457316 32682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:46517 every 8 connection(s)
I20251025 08:02:16.458192 32554 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[32306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:16.468235 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 32554
I20251025 08:02:16.468364 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:02:16.468628 28176 external_mini_cluster.cc:1468] Setting key 732c9d99e3308efbd40c8c9cba8d198b
I20251025 08:02:16.470049 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:16.459899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:55023 (local address 127.27.132.62:39153)
1025 08:02:16.460048 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:16.460052 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:16.460660 (+ 608us) server_negotiation.cc:408] Connection header received
1025 08:02:16.461428 (+ 768us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:16.461431 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:16.461478 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:16.461552 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:16.463128 (+ 1576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.463665 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:16.464265 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.464396 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:16.466483 (+ 2087us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:16.466502 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:16.466505 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:16.466530 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:16.468053 (+ 1523us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:16.468601 (+ 548us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:16.468604 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:16.468606 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:16.468664 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:16.468958 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:16.468961 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:16.468963 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:16.469131 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:16.469214 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:16.469770 (+ 556us) server_negotiation.cc:300] Negotiation successful
1025 08:02:16.469889 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251025 08:02:16.470834 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:16.460111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:39153 (local address 127.27.132.2:55023)
1025 08:02:16.460510 (+ 399us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:16.460545 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:16.461246 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:16.461717 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:16.461724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:16.462095 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:16.463008 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:16.463021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.463768 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:16.463771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:16.464168 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:16.464174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.464341 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:16.464908 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:16.464929 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:16.466322 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:16.468185 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:16.468190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:16.468199 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:16.468494 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:16.468766 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:16.468769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:16.468772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:16.468863 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:16.469227 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:16.469234 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:16.469534 (+ 300us) client_negotiation.cc:769] Sending connection context
1025 08:02:16.469714 (+ 180us) client_negotiation.cc:240] Negotiation successful
1025 08:02:16.469984 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20251025 08:02:16.472313 32683 heartbeater.cc:344] Connected to a master server at 127.27.132.62:39153
I20251025 08:02:16.472570 32683 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:16.473090 32683 heartbeater.cc:507] Master 127.27.132.62:39153 requested a full tablet report, sending...
I20251025 08:02:16.474192 32354 ts_manager.cc:194] Registered new tserver with Master: 90c635c3b56942c0a34728181fa1b737 (127.27.132.2:46517)
I20251025 08:02:16.474756 32354 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:55023
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:16.518565 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:39153
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:16.626979 32690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:16.627211 32690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:16.627271 32690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:16.630556 32690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:16.630633 32690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:16.630730 32690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:02:16.635553 32690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:39153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:16.636639 32690 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:16.637492 32690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:16.643939 32696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:16.643965 32695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:16.643975 32698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:16.644552 32690 server_base.cc:1047] running on GCE node
I20251025 08:02:16.644886 32690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:16.645443 32690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:16.646598 32690 hybrid_clock.cc:648] HybridClock initialized: now 1761379336646574 us; error 41 us; skew 500 ppm
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[32306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:16.649250 32690 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:02:16.650310 32690 webserver.cc:492] Webserver started at http://127.27.132.3:37895/ using document root <none> and password file <none>
I20251025 08:02:16.650866 32690 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:16.650913 32690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:16.651067 32690 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:16.652758 32690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "8f7e5605727f4ce1ba75648dc75944a9"
format_stamp: "Formatted at 2025-10-25 08:02:16 on dist-test-slave-lq8l"
server_key: "019970fe6af1a644df5f6f00817d4e8b"
server_key_iv: "c6e705c641cb6b8c8f4e854f556f5c23"
server_key_version: "encryptionkey@0"
I20251025 08:02:16.653187 32690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "8f7e5605727f4ce1ba75648dc75944a9"
format_stamp: "Formatted at 2025-10-25 08:02:16 on dist-test-slave-lq8l"
server_key: "019970fe6af1a644df5f6f00817d4e8b"
server_key_iv: "c6e705c641cb6b8c8f4e854f556f5c23"
server_key_version: "encryptionkey@0"
I20251025 08:02:16.656543 32690 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:02:16.658682 32705 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:16.659812 32690 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:16.659905 32690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "8f7e5605727f4ce1ba75648dc75944a9"
format_stamp: "Formatted at 2025-10-25 08:02:16 on dist-test-slave-lq8l"
server_key: "019970fe6af1a644df5f6f00817d4e8b"
server_key_iv: "c6e705c641cb6b8c8f4e854f556f5c23"
server_key_version: "encryptionkey@0"
I20251025 08:02:16.659982 32690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:16.669706 32690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:16.672644 32690 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:16.672807 32690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:16.673283 32690 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:16.674103 32690 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:16.674157 32690 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:16.674202 32690 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:16.674242 32690 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:16.684039 32690 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:33591
I20251025 08:02:16.684095 350 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:33591 every 8 connection(s)
I20251025 08:02:16.684978 32690 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[32306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:16.694823 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 32690
I20251025 08:02:16.694932 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:02:16.695168 28176 external_mini_cluster.cc:1468] Setting key 2bb35ad440db8c6ef575452aab5764a1
I20251025 08:02:16.697415 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:16.686653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:45915 (local address 127.27.132.62:39153)
1025 08:02:16.686845 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:16.686850 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:16.687543 (+ 693us) server_negotiation.cc:408] Connection header received
1025 08:02:16.688377 (+ 834us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:16.688381 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:16.688434 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:16.688517 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:16.690179 (+ 1662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.690736 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:16.691413 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.691551 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:16.693735 (+ 2184us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:16.693757 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:16.693760 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:16.693786 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:16.695350 (+ 1564us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:16.696016 (+ 666us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:16.696019 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:16.696020 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:16.696066 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:16.696356 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:16.696360 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:16.696362 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:16.696527 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:16.696622 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:16.697134 (+ 512us) server_negotiation.cc:300] Negotiation successful
1025 08:02:16.697268 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20251025 08:02:16.698071 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:16.686924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:39153 (local address 127.27.132.3:45915)
1025 08:02:16.687389 (+ 465us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:16.687429 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:16.688146 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:16.688672 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:16.688680 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:16.689066 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:16.690015 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:16.690030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.690845 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:16.690848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:16.691244 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:16.691250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.691480 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:16.692101 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:16.692126 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:16.693575 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:16.695494 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:16.695500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:16.695513 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:16.695774 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:16.696163 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:16.696167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:16.696169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:16.696261 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:16.696634 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:16.696641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:16.696884 (+ 243us) client_negotiation.cc:769] Sending connection context
1025 08:02:16.697076 (+ 192us) client_negotiation.cc:240] Negotiation successful
1025 08:02:16.697365 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20251025 08:02:16.699195 351 heartbeater.cc:344] Connected to a master server at 127.27.132.62:39153
I20251025 08:02:16.699496 351 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:16.700023 351 heartbeater.cc:507] Master 127.27.132.62:39153 requested a full tablet report, sending...
I20251025 08:02:16.701120 32354 ts_manager.cc:194] Registered new tserver with Master: 8f7e5605727f4ce1ba75648dc75944a9 (127.27.132.3:33591)
I20251025 08:02:16.701669 32354 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:45915
I20251025 08:02:16.709450 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[32306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[32306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.1@KRBTEST.COM
I20251025 08:02:16.733835 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:16.725195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.27.132.1:36537)
1025 08:02:16.725512 (+ 317us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:16.725522 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:16.725553 (+ 31us) server_negotiation.cc:408] Connection header received
1025 08:02:16.725611 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:16.725616 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:16.725786 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:16.725929 (+ 143us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:16.726752 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.727520 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:16.728197 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.728468 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:16.730420 (+ 1952us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:16.730445 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:16.730464 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:16.730488 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:16.732233 (+ 1745us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:16.732645 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:16.732650 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:16.732651 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:16.732706 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:16.732938 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:16.732942 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:16.732944 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:16.733125 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:16.733317 (+ 192us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:16.733521 (+ 204us) server_negotiation.cc:300] Negotiation successful
1025 08:02:16.733673 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[32306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:16.743676 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:16.736602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.27.132.62:39153)
1025 08:02:16.736747 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:16.736751 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:16.736823 (+ 72us) server_negotiation.cc:408] Connection header received
1025 08:02:16.736911 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:16.736913 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:16.736957 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:16.737022 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:16.737653 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.738140 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:16.738682 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:16.738808 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:16.740941 (+ 2133us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:16.740959 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:16.740961 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:16.740986 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:16.742205 (+ 1219us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:16.742580 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:16.742584 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:16.742585 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:16.742627 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:16.742842 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:16.742846 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:16.742848 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:16.743024 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:16.743149 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:16.743398 (+ 249us) server_negotiation.cc:300] Negotiation successful
1025 08:02:16.743513 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20251025 08:02:16.744097 32354 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40982
I20251025 08:02:16.744727 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 32418
I20251025 08:02:16.750659 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 32554
I20251025 08:02:16.756229 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 32690
I20251025 08:02:16.761477 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 32322
2025-10-25T08:02:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3119 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[363](info): set up 2 sockets
Oct 25 08:02:16 dist-test-slave-lq8l krb5kdc[363](info): commencing operation
krb5kdc: starting...
W20251025 08:02:18.756729 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.973s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:02:18 dist-test-slave-lq8l krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:02:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:18.903983 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:43529
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:43529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:19.021340 379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:19.021612 379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:19.021677 379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:19.025134 379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:19.025206 379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:19.025230 379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:19.025250 379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:19.025267 379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:19.029778 379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:43529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:43529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:19.031011 379 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:19.032099 379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:19.038103 387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:19.038103 385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:19.038165 379 server_base.cc:1047] running on GCE node
W20251025 08:02:19.038105 384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:19.038857 379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:19.039912 379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:19.041105 379 hybrid_clock.cc:648] HybridClock initialized: now 1761379339041101 us; error 40 us; skew 500 ppm
Oct 25 08:02:19 dist-test-slave-lq8l krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379339, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:19.044252 379 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:02:19.045042 379 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:19.045111 379 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:19.045320 379 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:19.047230 379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "1fcbc7b546594b28b83b767ad054a2e7"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "235fd8deb7b03323698602ebbca2cfc0"
server_key_iv: "0c972d017ea4a0fe954efc757c021bc0"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.047796 379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "1fcbc7b546594b28b83b767ad054a2e7"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "235fd8deb7b03323698602ebbca2cfc0"
server_key_iv: "0c972d017ea4a0fe954efc757c021bc0"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.051294 379 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251025 08:02:19.053797 393 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:19.054857 379 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:19.054982 379 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "1fcbc7b546594b28b83b767ad054a2e7"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "235fd8deb7b03323698602ebbca2cfc0"
server_key_iv: "0c972d017ea4a0fe954efc757c021bc0"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.055087 379 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:19.072865 379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:19.076218 379 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:19.076540 379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:19.084842 379 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:43529
I20251025 08:02:19.084836 445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:43529 every 8 connection(s)
I20251025 08:02:19.085558 379 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:19.088459 446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:19.090708 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 379
I20251025 08:02:19.090854 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:19.091109 28176 external_mini_cluster.cc:1468] Setting key 0975f2f49d9a190943ac28c19688e5ea
I20251025 08:02:19.095088 446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7: Bootstrap starting.
I20251025 08:02:19.097481 446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:19.098217 446 log.cc:826] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7: Log is configured to *not* fsync() on all Append() calls
Oct 25 08:02:19 dist-test-slave-lq8l krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:19.100548 446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7: No bootstrap required, opened a new log
I20251025 08:02:19.103222 446 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcbc7b546594b28b83b767ad054a2e7" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 43529 } }
I20251025 08:02:19.103530 446 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:19.103616 446 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcbc7b546594b28b83b767ad054a2e7, State: Initialized, Role: FOLLOWER
I20251025 08:02:19.104048 446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcbc7b546594b28b83b767ad054a2e7" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 43529 } }
I20251025 08:02:19.104168 446 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:19.104243 446 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:19.104271 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.092571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.27.132.62:43529)
1025 08:02:19.093022 (+ 451us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:19.093032 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:19.093067 (+ 35us) server_negotiation.cc:408] Connection header received
1025 08:02:19.093619 (+ 552us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:19.093638 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:19.093941 (+ 303us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:19.094261 (+ 320us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:19.095230 (+ 969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.096084 (+ 854us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.096747 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.097049 (+ 302us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.099448 (+ 2399us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:19.099467 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:19.099479 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:19.099514 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:19.101584 (+ 2070us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:19.102044 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:19.102049 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:19.102054 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:19.102131 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:19.102362 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:19.102365 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:19.102367 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:19.102729 (+ 362us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:19.102932 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:19.103222 (+ 290us) server_negotiation.cc:300] Negotiation successful
1025 08:02:19.103527 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":148,"threads_started":1}
I20251025 08:02:19.104331 446 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:19.105191 446 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcbc7b546594b28b83b767ad054a2e7" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 43529 } }
I20251025 08:02:19.105566 446 leader_election.cc:304] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fcbc7b546594b28b83b767ad054a2e7; no voters:
I20251025 08:02:19.105908 446 leader_election.cc:290] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:19.106374 451 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:19.106627 451 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1fcbc7b546594b28b83b767ad054a2e7, State: Running, Role: LEADER
I20251025 08:02:19.107003 451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcbc7b546594b28b83b767ad054a2e7" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 43529 } }
I20251025 08:02:19.107055 446 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:19.108409 452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fcbc7b546594b28b83b767ad054a2e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcbc7b546594b28b83b767ad054a2e7" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 43529 } } }
I20251025 08:02:19.108515 452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:19.108851 453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fcbc7b546594b28b83b767ad054a2e7. Latest consensus state: current_term: 1 leader_uuid: "1fcbc7b546594b28b83b767ad054a2e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcbc7b546594b28b83b767ad054a2e7" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 43529 } } }
I20251025 08:02:19.108945 453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7 [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:19.108909 460 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:19.112434 460 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:19.117540 460 catalog_manager.cc:1357] Generated new cluster ID: bfdb9940d46b4c5fb6534c8fa04fd4aa
I20251025 08:02:19.117630 460 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:19.132566 460 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:19.133958 460 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:19.144260 460 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7: Generated new TSK 0
I20251025 08:02:19.146066 460 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:19.196439 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:43529
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:19.312103 474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:19.312348 474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:19.312439 474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:19.315795 474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:19.315874 474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:19.315997 474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:02:19.320632 474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:19.321890 474 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:19.322969 474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:19.330569 482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:19.330569 480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:19.330621 474 server_base.cc:1047] running on GCE node
W20251025 08:02:19.330569 479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:19.331277 474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:19.331955 474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:19.333133 474 hybrid_clock.cc:648] HybridClock initialized: now 1761379339333116 us; error 33 us; skew 500 ppm
Oct 25 08:02:19 dist-test-slave-lq8l krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379339, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:19.336038 474 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:02:19.336876 474 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:19.336948 474 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:19.337150 474 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:19.338980 474 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "812efe1b7f1440e2a505ba7a7f602d84"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "0e118c258391a284fccd97a26d59a88c"
server_key_iv: "d003fc3c1d3faccec959130e964be017"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.339536 474 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "812efe1b7f1440e2a505ba7a7f602d84"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "0e118c258391a284fccd97a26d59a88c"
server_key_iv: "d003fc3c1d3faccec959130e964be017"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.343079 474 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20251025 08:02:19.345558 488 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:19.346741 474 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:19.346866 474 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "812efe1b7f1440e2a505ba7a7f602d84"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "0e118c258391a284fccd97a26d59a88c"
server_key_iv: "d003fc3c1d3faccec959130e964be017"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.346978 474 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:19.372649 474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:19.376226 474 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:19.376446 474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:19.377070 474 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:19.378010 474 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:19.378062 474 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:19.378123 474 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:19.378157 474 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:19.387986 474 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:46453
I20251025 08:02:19.388049 601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:46453 every 8 connection(s)
I20251025 08:02:19.388634 474 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:02:19.393472 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 474
I20251025 08:02:19.393589 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:02:19.393844 28176 external_mini_cluster.cc:1468] Setting key 243ba60fa9bb88aed6e7bd88477382a6
Oct 25 08:02:19 dist-test-slave-lq8l krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379339, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:19.402434 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.390487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:54371 (local address 127.27.132.62:43529)
1025 08:02:19.390643 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:19.390647 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:19.391394 (+ 747us) server_negotiation.cc:408] Connection header received
1025 08:02:19.392310 (+ 916us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:19.392314 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:19.392378 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:19.392456 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:19.394308 (+ 1852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.394844 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.395594 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.395785 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.398639 (+ 2854us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:19.398662 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:19.398664 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:19.398693 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:19.400405 (+ 1712us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:19.400935 (+ 530us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:19.400938 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:19.400940 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:19.400990 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:19.401281 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:19.401285 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:19.401287 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:19.401455 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:19.401556 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:19.402101 (+ 545us) server_negotiation.cc:300] Negotiation successful
1025 08:02:19.402251 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
I20251025 08:02:19.403142 604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.390779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:43529 (local address 127.27.132.1:54371)
1025 08:02:19.391238 (+ 459us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:19.391276 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:19.392094 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:19.392657 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:19.392666 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:19.393099 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:19.394140 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.394157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.394983 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.394986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:19.395465 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.395474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.395690 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.396500 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:19.396527 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:19.398401 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:19.400544 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:19.400550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:19.400562 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:19.400824 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:19.401081 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:19.401084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:19.401086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:19.401188 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:19.401576 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:19.401584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:19.401848 (+ 264us) client_negotiation.cc:769] Sending connection context
1025 08:02:19.402050 (+ 202us) client_negotiation.cc:240] Negotiation successful
1025 08:02:19.402353 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":129,"threads_started":1}
I20251025 08:02:19.404439 602 heartbeater.cc:344] Connected to a master server at 127.27.132.62:43529
I20251025 08:02:19.404717 602 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:19.405230 602 heartbeater.cc:507] Master 127.27.132.62:43529 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
I20251025 08:02:19.406870 406 ts_manager.cc:194] Registered new tserver with Master: 812efe1b7f1440e2a505ba7a7f602d84 (127.27.132.1:46453)
I20251025 08:02:19.408700 406 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:54371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:19.449824 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:43529
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:19.562255 609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:19.562508 609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:19.562572 609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:19.565910 609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:19.565984 609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:19.566071 609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:02:19.570521 609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:19.571681 609 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:19.572525 609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:19.579288 617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:19.579289 614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:19.579306 615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:19.579612 609 server_base.cc:1047] running on GCE node
I20251025 08:02:19.580096 609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:19.580711 609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:19.581871 609 hybrid_clock.cc:648] HybridClock initialized: now 1761379339581853 us; error 43 us; skew 500 ppm
Oct 25 08:02:19 dist-test-slave-lq8l krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379339, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:19.584800 609 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:02:19.585620 609 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:19.585692 609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:19.585897 609 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:19.587759 609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "4d37716173fb4bf68e2efe17fdfcdc23"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "194272974e9e2a9040046471f83ef332"
server_key_iv: "c64af68745083b143a65a70843bc0c77"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.588248 609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "4d37716173fb4bf68e2efe17fdfcdc23"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "194272974e9e2a9040046471f83ef332"
server_key_iv: "c64af68745083b143a65a70843bc0c77"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.591851 609 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:02:19.594204 623 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:19.595273 609 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:19.595436 609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "4d37716173fb4bf68e2efe17fdfcdc23"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "194272974e9e2a9040046471f83ef332"
server_key_iv: "c64af68745083b143a65a70843bc0c77"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.595554 609 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:19.613981 609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:19.617007 609 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:19.617204 609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:19.617774 609 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:19.618685 609 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:19.618738 609 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:19.618801 609 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:19.618830 609 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:19.628422 609 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:33059
I20251025 08:02:19.628443 736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:33059 every 8 connection(s)
I20251025 08:02:19.629026 609 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:02:19.636341 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 609
I20251025 08:02:19.636482 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:02:19.636804 28176 external_mini_cluster.cc:1468] Setting key 336858bd64b400ba6a2e4e5bd214d918
Oct 25 08:02:19 dist-test-slave-lq8l krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379339, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:19.643249 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.630798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:48927 (local address 127.27.132.62:43529)
1025 08:02:19.630939 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:19.630942 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:19.631671 (+ 729us) server_negotiation.cc:408] Connection header received
1025 08:02:19.632745 (+ 1074us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:19.632749 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:19.632817 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:19.632896 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:19.634651 (+ 1755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.635164 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.635901 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.636081 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.638907 (+ 2826us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:19.638929 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:19.638933 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:19.638969 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:19.640717 (+ 1748us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:19.641414 (+ 697us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:19.641416 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:19.641418 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:19.641470 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:19.641815 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:19.641818 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:19.641819 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:19.641980 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:19.642055 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:19.642877 (+ 822us) server_negotiation.cc:300] Negotiation successful
1025 08:02:19.643005 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 08:02:19.644016 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.631024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:43529 (local address 127.27.132.2:48927)
1025 08:02:19.631518 (+ 494us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:19.631557 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:19.632472 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:19.633084 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:19.633093 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:19.633513 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:19.634475 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.634491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.635297 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.635300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:19.635761 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.635769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.635990 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.636845 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:19.636904 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:19.638712 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:19.640876 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:19.640884 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:19.640900 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:19.641296 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:19.641553 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:19.641559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:19.641564 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:19.641724 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:19.642084 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:19.642095 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:19.642503 (+ 408us) client_negotiation.cc:769] Sending connection context
1025 08:02:19.642813 (+ 310us) client_negotiation.cc:240] Negotiation successful
1025 08:02:19.643146 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":121,"threads_started":1}
I20251025 08:02:19.645264 737 heartbeater.cc:344] Connected to a master server at 127.27.132.62:43529
I20251025 08:02:19.645555 737 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:19.646099 737 heartbeater.cc:507] Master 127.27.132.62:43529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
I20251025 08:02:19.647233 406 ts_manager.cc:194] Registered new tserver with Master: 4d37716173fb4bf68e2efe17fdfcdc23 (127.27.132.2:33059)
I20251025 08:02:19.647881 406 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:48927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:19.690601 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:43529
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:19.801594 744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:19.801846 744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:19.801944 744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:19.805362 744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:19.805593 744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:19.805734 744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:02:19.810438 744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:19.811724 744 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:19.812628 744 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:19.819443 750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:19.819450 752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:19.819502 749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:19.819902 744 server_base.cc:1047] running on GCE node
I20251025 08:02:19.820307 744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:19.820915 744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:19.822105 744 hybrid_clock.cc:648] HybridClock initialized: now 1761379339822077 us; error 45 us; skew 500 ppm
Oct 25 08:02:19 dist-test-slave-lq8l krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379339, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:19.825070 744 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:02:19.825896 744 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:19.825965 744 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:19.826170 744 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:19.828055 744 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "80d1b90f11db436b88a5bbff0e02437b"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "fa284adb200caa8eeb8bb83249ce5faf"
server_key_iv: "2484f0832f382931b7487e4e95c52515"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.828550 744 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "80d1b90f11db436b88a5bbff0e02437b"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "fa284adb200caa8eeb8bb83249ce5faf"
server_key_iv: "2484f0832f382931b7487e4e95c52515"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.832346 744 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251025 08:02:19.834837 758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:19.835999 744 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:19.836134 744 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "80d1b90f11db436b88a5bbff0e02437b"
format_stamp: "Formatted at 2025-10-25 08:02:19 on dist-test-slave-lq8l"
server_key: "fa284adb200caa8eeb8bb83249ce5faf"
server_key_iv: "2484f0832f382931b7487e4e95c52515"
server_key_version: "encryptionkey@0"
I20251025 08:02:19.836241 744 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:19.870440 744 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:19.873735 744 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:19.873939 744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:19.874536 744 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:19.875540 744 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:19.875592 744 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:19.875634 744 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:19.875730 744 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:19.885303 744 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:44053
I20251025 08:02:19.885330 871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:44053 every 8 connection(s)
I20251025 08:02:19.885924 744 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:02:19.887504 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 744
I20251025 08:02:19.887656 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:02:19.887962 28176 external_mini_cluster.cc:1468] Setting key d00260f10a2680a4c1a1921863e47585
Oct 25 08:02:19 dist-test-slave-lq8l krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379339, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:19.899096 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.887760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:41211 (local address 127.27.132.62:43529)
1025 08:02:19.887931 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:19.887937 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:19.888821 (+ 884us) server_negotiation.cc:408] Connection header received
1025 08:02:19.889806 (+ 985us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:19.889810 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:19.889871 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:19.889947 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:19.891771 (+ 1824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.892273 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.892966 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.893148 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.895732 (+ 2584us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:19.895754 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:19.895756 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:19.895785 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:19.897124 (+ 1339us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:19.897649 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:19.897652 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:19.897653 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:19.897700 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:19.897982 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:19.897988 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:19.897990 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:19.898152 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:19.898247 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:19.898775 (+ 528us) server_negotiation.cc:300] Negotiation successful
1025 08:02:19.898908 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251025 08:02:19.899833 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.888107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:43529 (local address 127.27.132.3:41211)
1025 08:02:19.888683 (+ 576us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:19.888724 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:19.889600 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:19.890125 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:19.890135 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:19.890578 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:19.891606 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.891623 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.892413 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.892416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:19.892841 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.892849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.893075 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.893808 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:19.893835 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:19.895551 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:19.897261 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:19.897267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:19.897281 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:19.897534 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:19.897795 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:19.897798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:19.897799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:19.897889 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:19.898261 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:19.898271 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:19.898525 (+ 254us) client_negotiation.cc:769] Sending connection context
1025 08:02:19.898724 (+ 199us) client_negotiation.cc:240] Negotiation successful
1025 08:02:19.899016 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":160,"threads_started":1}
I20251025 08:02:19.901014 872 heartbeater.cc:344] Connected to a master server at 127.27.132.62:43529
I20251025 08:02:19.901280 872 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:19.901927 872 heartbeater.cc:507] Master 127.27.132.62:43529 requested a full tablet report, sending...
I20251025 08:02:19.902896 406 ts_manager.cc:194] Registered new tserver with Master: 80d1b90f11db436b88a5bbff0e02437b (127.27.132.3:44053)
I20251025 08:02:19.903513 406 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:41211
I20251025 08:02:19.913889 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 08:02:19.922860 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.915918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.27.132.62:43529)
1025 08:02:19.916115 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:19.916120 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:19.916179 (+ 59us) server_negotiation.cc:408] Connection header received
1025 08:02:19.916346 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:19.916349 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:19.916425 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:19.916504 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:19.917409 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.917905 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.918505 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.918643 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.919834 (+ 1191us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:19.919853 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:19.919855 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:19.919881 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:19.921549 (+ 1668us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:19.921930 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:19.921933 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:19.921935 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:19.921973 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:19.922137 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:19.922139 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:19.922140 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:19.922314 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:19.922414 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:19.922572 (+ 158us) server_negotiation.cc:300] Negotiation successful
1025 08:02:19.922671 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20251025 08:02:19.925984 406 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:02:19.928427 406 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:02:19.941874 888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.937530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:33059 (local address 127.0.0.1:49976)
1025 08:02:19.938163 (+ 633us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:19.938176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:19.938249 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:19.938885 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:19.938888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:19.938904 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:19.939137 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.939146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.940585 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.940588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:19.941410 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.941422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.941529 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.941579 (+ 50us) client_negotiation.cc:769] Sending connection context
1025 08:02:19.941662 (+ 83us) client_negotiation.cc:240] Negotiation successful
1025 08:02:19.941740 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":105,"threads_started":1}
I20251025 08:02:19.941915 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.936978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:46453 (local address 127.0.0.1:37138)
1025 08:02:19.937549 (+ 571us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:19.937592 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:19.937752 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:19.938723 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:19.938726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:19.938743 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:19.939019 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.939027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.940428 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.940432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:19.941298 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.941307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.941489 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.941565 (+ 76us) client_negotiation.cc:769] Sending connection context
1025 08:02:19.941669 (+ 104us) client_negotiation.cc:240] Negotiation successful
1025 08:02:19.941754 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":100,"threads_started":1}
I20251025 08:02:19.942195 886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.936958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39582 (local address 127.27.132.3:44053)
1025 08:02:19.937563 (+ 605us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:19.937573 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:19.937787 (+ 214us) server_negotiation.cc:408] Connection header received
1025 08:02:19.938020 (+ 233us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:19.938028 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:19.938233 (+ 205us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:19.938359 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:19.939075 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.939902 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.941063 (+ 1161us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.941598 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.941730 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:19.941959 (+ 229us) server_negotiation.cc:300] Negotiation successful
1025 08:02:19.942068 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":130,"threads_started":1}
I20251025 08:02:19.942039 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.936794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:44053 (local address 127.0.0.1:39582)
1025 08:02:19.937694 (+ 900us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:19.937709 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:19.937859 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:19.938580 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:19.938588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:19.938625 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:19.938934 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.938943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.940035 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.940038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:19.940895 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:19.940905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.941788 (+ 883us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.941803 (+ 15us) client_negotiation.cc:769] Sending connection context
1025 08:02:19.941878 (+ 75us) client_negotiation.cc:240] Negotiation successful
1025 08:02:19.941934 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":91,"threads_started":1}
I20251025 08:02:19.943039 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.937439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.27.132.1:46453)
1025 08:02:19.937734 (+ 295us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:19.937742 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:19.937771 (+ 29us) server_negotiation.cc:408] Connection header received
1025 08:02:19.937997 (+ 226us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:19.938007 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:19.938581 (+ 574us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:19.938730 (+ 149us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:19.939152 (+ 422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.940295 (+ 1143us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.942025 (+ 1730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.942610 (+ 585us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.942733 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:19.942806 (+ 73us) server_negotiation.cc:300] Negotiation successful
1025 08:02:19.942902 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
I20251025 08:02:19.943230 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:19.937942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.27.132.2:33059)
1025 08:02:19.938359 (+ 417us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:19.938367 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:19.938392 (+ 25us) server_negotiation.cc:408] Connection header received
1025 08:02:19.938475 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:19.938480 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:19.938600 (+ 120us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:19.938736 (+ 136us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:19.939277 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.940456 (+ 1179us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:19.942154 (+ 1698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:19.942836 (+ 682us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:19.942933 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:19.942999 (+ 66us) server_negotiation.cc:300] Negotiation successful
1025 08:02:19.943088 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":100,"threads_started":1}
I20251025 08:02:19.945871 806 tablet_service.cc:1505] Processing CreateTablet for tablet 04b7ec8e104e444a969ef9345ba78659 (DEFAULT_TABLE table=test-table [id=ebdf04eac01441abac637c54148530a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:19.945899 536 tablet_service.cc:1505] Processing CreateTablet for tablet 04b7ec8e104e444a969ef9345ba78659 (DEFAULT_TABLE table=test-table [id=ebdf04eac01441abac637c54148530a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:19.945911 671 tablet_service.cc:1505] Processing CreateTablet for tablet 04b7ec8e104e444a969ef9345ba78659 (DEFAULT_TABLE table=test-table [id=ebdf04eac01441abac637c54148530a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:19.946802 806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b7ec8e104e444a969ef9345ba78659. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:19.946810 671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b7ec8e104e444a969ef9345ba78659. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:19.946810 536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b7ec8e104e444a969ef9345ba78659. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:19.952621 890 tablet_bootstrap.cc:492] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: Bootstrap starting.
I20251025 08:02:19.953868 891 tablet_bootstrap.cc:492] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84: Bootstrap starting.
I20251025 08:02:19.955353 890 tablet_bootstrap.cc:654] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:19.955663 891 tablet_bootstrap.cc:654] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:19.955729 892 tablet_bootstrap.cc:492] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: Bootstrap starting.
I20251025 08:02:19.956251 890 log.cc:826] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:19.956337 891 log.cc:826] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:19.957969 892 tablet_bootstrap.cc:654] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:19.958511 891 tablet_bootstrap.cc:492] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84: No bootstrap required, opened a new log
I20251025 08:02:19.958730 891 ts_tablet_manager.cc:1403] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 08:02:19.958799 892 log.cc:826] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:19.959174 890 tablet_bootstrap.cc:492] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: No bootstrap required, opened a new log
I20251025 08:02:19.959390 890 ts_tablet_manager.cc:1403] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20251025 08:02:19.961509 891 raft_consensus.cc:359] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d1b90f11db436b88a5bbff0e02437b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 44053 } } peers { permanent_uuid: "812efe1b7f1440e2a505ba7a7f602d84" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46453 } } peers { permanent_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 33059 } }
I20251025 08:02:19.961711 891 raft_consensus.cc:385] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:19.961755 891 raft_consensus.cc:740] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 812efe1b7f1440e2a505ba7a7f602d84, State: Initialized, Role: FOLLOWER
I20251025 08:02:19.962188 891 consensus_queue.cc:260] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d1b90f11db436b88a5bbff0e02437b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 44053 } } peers { permanent_uuid: "812efe1b7f1440e2a505ba7a7f602d84" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46453 } } peers { permanent_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 33059 } }
I20251025 08:02:19.962520 890 raft_consensus.cc:359] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d1b90f11db436b88a5bbff0e02437b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 44053 } } peers { permanent_uuid: "812efe1b7f1440e2a505ba7a7f602d84" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46453 } } peers { permanent_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 33059 } }
I20251025 08:02:19.962746 890 raft_consensus.cc:385] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:19.962813 890 raft_consensus.cc:740] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d1b90f11db436b88a5bbff0e02437b, State: Initialized, Role: FOLLOWER
I20251025 08:02:19.962934 602 heartbeater.cc:499] Master 127.27.132.62:43529 was elected leader, sending a full tablet report...
I20251025 08:02:19.963076 891 ts_tablet_manager.cc:1434] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20251025 08:02:19.963280 890 consensus_queue.cc:260] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d1b90f11db436b88a5bbff0e02437b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 44053 } } peers { permanent_uuid: "812efe1b7f1440e2a505ba7a7f602d84" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46453 } } peers { permanent_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 33059 } }
I20251025 08:02:19.964063 872 heartbeater.cc:499] Master 127.27.132.62:43529 was elected leader, sending a full tablet report...
I20251025 08:02:19.964329 890 ts_tablet_manager.cc:1434] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251025 08:02:19.965054 892 tablet_bootstrap.cc:492] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: No bootstrap required, opened a new log
I20251025 08:02:19.965265 892 ts_tablet_manager.cc:1403] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20251025 08:02:19.968912 892 raft_consensus.cc:359] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d1b90f11db436b88a5bbff0e02437b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 44053 } } peers { permanent_uuid: "812efe1b7f1440e2a505ba7a7f602d84" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46453 } } peers { permanent_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 33059 } }
I20251025 08:02:19.969110 892 raft_consensus.cc:385] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:19.969146 892 raft_consensus.cc:740] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d37716173fb4bf68e2efe17fdfcdc23, State: Initialized, Role: FOLLOWER
I20251025 08:02:19.969506 892 consensus_queue.cc:260] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d1b90f11db436b88a5bbff0e02437b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 44053 } } peers { permanent_uuid: "812efe1b7f1440e2a505ba7a7f602d84" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46453 } } peers { permanent_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 33059 } }
I20251025 08:02:19.970176 737 heartbeater.cc:499] Master 127.27.132.62:43529 was elected leader, sending a full tablet report...
I20251025 08:02:19.970327 892 ts_tablet_manager.cc:1434] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20251025 08:02:20.130081 738 tablet.cc:2378] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251025 08:02:20.136803 873 tablet.cc:2378] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251025 08:02:20.139935 603 tablet.cc:2378] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 08:02:20.228188 898 raft_consensus.cc:493] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:02:20.228415 898 raft_consensus.cc:515] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d1b90f11db436b88a5bbff0e02437b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 44053 } } peers { permanent_uuid: "812efe1b7f1440e2a505ba7a7f602d84" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46453 } } peers { permanent_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 33059 } }
I20251025 08:02:20.229647 898 leader_election.cc:290] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80d1b90f11db436b88a5bbff0e02437b (127.27.132.3:44053), 812efe1b7f1440e2a505ba7a7f602d84 (127.27.132.1:46453)
I20251025 08:02:20.232717 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.229871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:44053 (local address 127.27.132.2:40625)
1025 08:02:20.230204 (+ 333us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:20.230216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:20.230326 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:20.230599 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:20.230602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:20.230619 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:20.230862 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:20.230868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:20.231650 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:20.231654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:20.232238 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:20.232246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:20.232365 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:20.232389 (+ 24us) client_negotiation.cc:769] Sending connection context
1025 08:02:20.232444 (+ 55us) client_negotiation.cc:240] Negotiation successful
1025 08:02:20.232514 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":75,"threads_started":1}
I20251025 08:02:20.232749 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.229871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:46453 (local address 127.27.132.2:34565)
1025 08:02:20.230312 (+ 441us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:20.230323 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:20.230445 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:20.230712 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:20.230714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:20.230731 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:20.230985 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:20.230991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:20.231742 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:20.231745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:20.232306 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:20.232313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:20.232425 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:20.232444 (+ 19us) client_negotiation.cc:769] Sending connection context
1025 08:02:20.232507 (+ 63us) client_negotiation.cc:240] Negotiation successful
1025 08:02:20.232566 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":44,"thread_start_us":91,"threads_started":1}
I20251025 08:02:20.233166 886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.229999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:40625 (local address 127.27.132.3:44053)
1025 08:02:20.230143 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:20.230149 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:20.230291 (+ 142us) server_negotiation.cc:408] Connection header received
1025 08:02:20.230453 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:20.230456 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:20.230501 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:20.230571 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:20.230976 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:20.231531 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:20.232375 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:20.232895 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:20.232942 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:20.232998 (+ 56us) server_negotiation.cc:300] Negotiation successful
1025 08:02:20.233065 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 08:02:20.233228 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.230034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:34565 (local address 127.27.132.1:46453)
1025 08:02:20.230256 (+ 222us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:20.230261 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:20.230428 (+ 167us) server_negotiation.cc:408] Connection header received
1025 08:02:20.230567 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:20.230570 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:20.230617 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:20.230687 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:20.231111 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:20.231617 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:20.232434 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:20.232934 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:20.232983 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:20.233039 (+ 56us) server_negotiation.cc:300] Negotiation successful
1025 08:02:20.233101 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20251025 08:02:20.233762 826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04b7ec8e104e444a969ef9345ba78659" candidate_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d1b90f11db436b88a5bbff0e02437b" is_pre_election: true
I20251025 08:02:20.234023 826 raft_consensus.cc:2468] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d37716173fb4bf68e2efe17fdfcdc23 in term 0.
I20251025 08:02:20.234040 556 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04b7ec8e104e444a969ef9345ba78659" candidate_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "812efe1b7f1440e2a505ba7a7f602d84" is_pre_election: true
I20251025 08:02:20.234279 556 raft_consensus.cc:2468] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d37716173fb4bf68e2efe17fdfcdc23 in term 0.
I20251025 08:02:20.234498 626 leader_election.cc:304] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d37716173fb4bf68e2efe17fdfcdc23, 80d1b90f11db436b88a5bbff0e02437b; no voters:
I20251025 08:02:20.234769 898 raft_consensus.cc:2804] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:02:20.234843 898 raft_consensus.cc:493] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:02:20.234894 898 raft_consensus.cc:3060] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:20.235920 898 raft_consensus.cc:515] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d1b90f11db436b88a5bbff0e02437b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 44053 } } peers { permanent_uuid: "812efe1b7f1440e2a505ba7a7f602d84" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46453 } } peers { permanent_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 33059 } }
I20251025 08:02:20.236279 898 leader_election.cc:290] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [CANDIDATE]: Term 1 election: Requested vote from peers 80d1b90f11db436b88a5bbff0e02437b (127.27.132.3:44053), 812efe1b7f1440e2a505ba7a7f602d84 (127.27.132.1:46453)
I20251025 08:02:20.236677 556 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04b7ec8e104e444a969ef9345ba78659" candidate_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "812efe1b7f1440e2a505ba7a7f602d84"
I20251025 08:02:20.236673 826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04b7ec8e104e444a969ef9345ba78659" candidate_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d1b90f11db436b88a5bbff0e02437b"
I20251025 08:02:20.236783 556 raft_consensus.cc:3060] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:20.236795 826 raft_consensus.cc:3060] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:20.237752 556 raft_consensus.cc:2468] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d37716173fb4bf68e2efe17fdfcdc23 in term 1.
I20251025 08:02:20.237780 826 raft_consensus.cc:2468] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d37716173fb4bf68e2efe17fdfcdc23 in term 1.
I20251025 08:02:20.238071 625 leader_election.cc:304] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d37716173fb4bf68e2efe17fdfcdc23, 812efe1b7f1440e2a505ba7a7f602d84; no voters:
I20251025 08:02:20.238265 898 raft_consensus.cc:2804] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:20.238490 898 raft_consensus.cc:697] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 1 LEADER]: Becoming Leader. State: Replica: 4d37716173fb4bf68e2efe17fdfcdc23, State: Running, Role: LEADER
I20251025 08:02:20.239013 898 consensus_queue.cc:237] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d1b90f11db436b88a5bbff0e02437b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 44053 } } peers { permanent_uuid: "812efe1b7f1440e2a505ba7a7f602d84" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46453 } } peers { permanent_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 33059 } }
I20251025 08:02:20.242245 406 catalog_manager.cc:5649] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d37716173fb4bf68e2efe17fdfcdc23 (127.27.132.2). New cstate: current_term: 1 leader_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80d1b90f11db436b88a5bbff0e02437b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 44053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "812efe1b7f1440e2a505ba7a7f602d84" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d37716173fb4bf68e2efe17fdfcdc23" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 33059 } health_report { overall_health: HEALTHY } } }
I20251025 08:02:20.307770 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.304576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.27.132.2:33059)
1025 08:02:20.304729 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:20.304733 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:20.304765 (+ 32us) server_negotiation.cc:408] Connection header received
1025 08:02:20.304847 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:20.304849 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:20.304906 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:20.304988 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:02:20.305292 (+ 304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:20.305806 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:20.306559 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:20.306739 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:20.306838 (+ 99us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:02:20.307296 (+ 458us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:02:20.307432 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:20.307528 (+ 96us) server_negotiation.cc:300] Negotiation successful
1025 08:02:20.307579 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251025 08:02:20.313417 556 raft_consensus.cc:1275] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84 [term 1 FOLLOWER]: Refusing update from remote peer 4d37716173fb4bf68e2efe17fdfcdc23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:20.313417 826 raft_consensus.cc:1275] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b [term 1 FOLLOWER]: Refusing update from remote peer 4d37716173fb4bf68e2efe17fdfcdc23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:20.314109 901 consensus_queue.cc:1048] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "812efe1b7f1440e2a505ba7a7f602d84" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:20.314301 898 consensus_queue.cc:1048] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d1b90f11db436b88a5bbff0e02437b" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 44053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:20.322017 906 mvcc.cc:204] Tried to move back new op lower bound from 7214609777916895232 to 7214609777621901312. Current Snapshot: MvccSnapshot[applied={T|T < 7214609777916895232}]
I20251025 08:02:20.323282 905 mvcc.cc:204] Tried to move back new op lower bound from 7214609777916895232 to 7214609777621901312. Current Snapshot: MvccSnapshot[applied={T|T < 7214609777916895232}]
I20251025 08:02:20.329182 406 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51674:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:20.329368 406 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51674:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:20.331877 406 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 1fcbc7b546594b28b83b767ad054a2e7: Sending DeleteTablet for 3 replicas of tablet 04b7ec8e104e444a969ef9345ba78659
I20251025 08:02:20.332801 536 tablet_service.cc:1552] Processing DeleteTablet for tablet 04b7ec8e104e444a969ef9345ba78659 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:20 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:37138
I20251025 08:02:20.332800 671 tablet_service.cc:1552] Processing DeleteTablet for tablet 04b7ec8e104e444a969ef9345ba78659 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:20 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:49976
I20251025 08:02:20.332800 806 tablet_service.cc:1552] Processing DeleteTablet for tablet 04b7ec8e104e444a969ef9345ba78659 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:20 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:39582
I20251025 08:02:20.333330 914 tablet_replica.cc:333] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84: stopping tablet replica
I20251025 08:02:20.333626 914 raft_consensus.cc:2243] T 04b7ec8e104e444a969ef9345ba78659 P 812efe1b7f1440e2a505ba7a7f602d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:20.333668 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 474
I20251025 08:02:20.335346 913 tablet_replica.cc:333] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: stopping tablet replica
I20251025 08:02:20.335623 913 raft_consensus.cc:2243] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:20.335953 913 raft_consensus.cc:2272] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:20.336328 915 tablet_replica.cc:333] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: stopping tablet replica
I20251025 08:02:20.336565 915 raft_consensus.cc:2243] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:02:20.337069 915 raft_consensus.cc:2272] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:20.337702 913 ts_tablet_manager.cc:1916] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:20.338572 915 ts_tablet_manager.cc:1916] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:20.341306 915 ts_tablet_manager.cc:1929] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:20.341504 915 log.cc:1199] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/04b7ec8e104e444a969ef9345ba78659
I20251025 08:02:20.341260 913 ts_tablet_manager.cc:1929] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:20.341579 913 log.cc:1199] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/04b7ec8e104e444a969ef9345ba78659
W20251025 08:02:20.341622 395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.132.1:46453 (error 108)
I20251025 08:02:20.341941 915 ts_tablet_manager.cc:1950] T 04b7ec8e104e444a969ef9345ba78659 P 4d37716173fb4bf68e2efe17fdfcdc23: Deleting consensus metadata
I20251025 08:02:20.342027 913 ts_tablet_manager.cc:1950] T 04b7ec8e104e444a969ef9345ba78659 P 80d1b90f11db436b88a5bbff0e02437b: Deleting consensus metadata
I20251025 08:02:20.342566 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 609
I20251025 08:02:20.343367 396 catalog_manager.cc:4985] TS 80d1b90f11db436b88a5bbff0e02437b (127.27.132.3:44053): tablet 04b7ec8e104e444a969ef9345ba78659 (table test-table [id=ebdf04eac01441abac637c54148530a5]) successfully deleted
I20251025 08:02:20.345077 888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.344766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:46453 (local address 127.0.0.1:37152)
1025 08:02:20.344917 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:20.344996 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.1:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20251025 08:02:20.345225 395 catalog_manager.cc:4712] TS 812efe1b7f1440e2a505ba7a7f602d84 (127.27.132.1:46453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04b7ec8e104e444a969ef9345ba78659: Network error: Client connection negotiation failed: client connection to 127.27.132.1:46453: connect: Connection refused (error 111)
I20251025 08:02:20.349612 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 744
I20251025 08:02:20.350678 888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.349855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:33059 (local address 127.0.0.1:49996)
1025 08:02:20.350497 (+ 642us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:20.350562 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.2:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519}
I20251025 08:02:20.355914 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 379
2025-10-25T08:02:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3594 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-10-25T08:02:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:20Z Disabled control of system clock
I20251025 08:02:20.386255 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:45065
--webserver_interface=127.27.132.62
--webserver_port=0
--builtin_ntp_servers=127.27.132.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:45065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251025 08:02:20.500025 920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:20.500290 920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:20.500342 920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:20.503793 920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:20.503876 920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:20.503893 920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:20.503912 920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:20.503930 920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:20.507997 920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:45065
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:45065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:20.509218 920 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:20.510442 920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:20.516258 928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:20.516258 925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:20.516342 920 server_base.cc:1047] running on GCE node
W20251025 08:02:20.516261 926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:20.517040 920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:20.517998 920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:20.519176 920 hybrid_clock.cc:648] HybridClock initialized: now 1761379340519166 us; error 29 us; skew 500 ppm
I20251025 08:02:20.521365 920 webserver.cc:492] Webserver started at http://127.27.132.62:34325/ using document root <none> and password file <none>
I20251025 08:02:20.521966 920 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:20.522029 920 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:20.522235 920 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:20.524082 920 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "b6e434f9ddf6476da5662cd0b3e3c17e"
format_stamp: "Formatted at 2025-10-25 08:02:20 on dist-test-slave-lq8l"
server_key: "7d116894af422d24d4ba996f40ebd063"
server_key_iv: "50e0f9004a176024b57c16071d3abc6b"
server_key_version: "encryptionkey@0"
I20251025 08:02:20.524593 920 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "b6e434f9ddf6476da5662cd0b3e3c17e"
format_stamp: "Formatted at 2025-10-25 08:02:20 on dist-test-slave-lq8l"
server_key: "7d116894af422d24d4ba996f40ebd063"
server_key_iv: "50e0f9004a176024b57c16071d3abc6b"
server_key_version: "encryptionkey@0"
I20251025 08:02:20.528239 920 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:02:20.530522 934 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:20.531793 920 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 08:02:20.531919 920 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "b6e434f9ddf6476da5662cd0b3e3c17e"
format_stamp: "Formatted at 2025-10-25 08:02:20 on dist-test-slave-lq8l"
server_key: "7d116894af422d24d4ba996f40ebd063"
server_key_iv: "50e0f9004a176024b57c16071d3abc6b"
server_key_version: "encryptionkey@0"
I20251025 08:02:20.532025 920 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:20.537066 920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:20.537675 920 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:20.537851 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:20.545121 986 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:45065 every 8 connection(s)
I20251025 08:02:20.545127 920 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:45065
I20251025 08:02:20.546193 920 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:20.549047 987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:20.551664 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 920
I20251025 08:02:20.551824 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:20.552129 28176 external_mini_cluster.cc:1468] Setting key 573b42be8568070efe90b3456ac1fa49
I20251025 08:02:20.555379 987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e: Bootstrap starting.
I20251025 08:02:20.557493 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.553521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.27.132.62:45065)
1025 08:02:20.554204 (+ 683us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:20.554214 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:20.554249 (+ 35us) server_negotiation.cc:408] Connection header received
1025 08:02:20.554759 (+ 510us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:20.554775 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:20.555097 (+ 322us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:20.555461 (+ 364us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:20.556335 (+ 874us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:20.556340 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:20.556351 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:20.556362 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:20.556405 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:20.556417 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:20.556559 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:20.556667 (+ 108us) server_negotiation.cc:300] Negotiation successful
1025 08:02:20.556919 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":145,"threads_started":1}
I20251025 08:02:20.558229 987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:20.559916 987 log.cc:826] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:20.561756 987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e: No bootstrap required, opened a new log
I20251025 08:02:20.564600 987 raft_consensus.cc:359] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e434f9ddf6476da5662cd0b3e3c17e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 45065 } }
I20251025 08:02:20.564810 987 raft_consensus.cc:385] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:20.564882 987 raft_consensus.cc:740] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e434f9ddf6476da5662cd0b3e3c17e, State: Initialized, Role: FOLLOWER
I20251025 08:02:20.565313 987 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e434f9ddf6476da5662cd0b3e3c17e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 45065 } }
I20251025 08:02:20.565443 987 raft_consensus.cc:399] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:20.565513 987 raft_consensus.cc:493] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:20.565613 987 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:20.566484 987 raft_consensus.cc:515] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e434f9ddf6476da5662cd0b3e3c17e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 45065 } }
I20251025 08:02:20.566833 987 leader_election.cc:304] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6e434f9ddf6476da5662cd0b3e3c17e; no voters:
I20251025 08:02:20.567122 987 leader_election.cc:290] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:20.567251 992 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:20.567516 992 raft_consensus.cc:697] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [term 1 LEADER]: Becoming Leader. State: Replica: b6e434f9ddf6476da5662cd0b3e3c17e, State: Running, Role: LEADER
I20251025 08:02:20.567864 992 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e434f9ddf6476da5662cd0b3e3c17e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 45065 } }
I20251025 08:02:20.568226 987 sys_catalog.cc:565] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:20.569509 993 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6e434f9ddf6476da5662cd0b3e3c17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e434f9ddf6476da5662cd0b3e3c17e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 45065 } } }
I20251025 08:02:20.569662 993 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:20.569509 994 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6e434f9ddf6476da5662cd0b3e3c17e. Latest consensus state: current_term: 1 leader_uuid: "b6e434f9ddf6476da5662cd0b3e3c17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e434f9ddf6476da5662cd0b3e3c17e" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 45065 } } }
I20251025 08:02:20.569741 994 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:20.570449 1001 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:20.573388 1001 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:20.578708 1001 catalog_manager.cc:1357] Generated new cluster ID: 448e58f681284d188f8146090ece1f37
I20251025 08:02:20.578876 1001 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:20.587147 1001 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:20.587499 1001 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:20.594157 1001 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e: Generated new TSK 0
I20251025 08:02:20.595067 1001 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 08:02:20.604180 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:45065
--builtin_ntp_servers=127.27.132.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251025 08:02:20.713424 1011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:20.713678 1011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:20.713783 1011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:20.717213 1011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:20.717283 1011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:20.717403 1011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:02:20.721357 1011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:20.722517 1011 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:20.723755 1011 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:20.730610 1017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:20.730726 1019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:20.730630 1016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:20.730947 1011 server_base.cc:1047] running on GCE node
I20251025 08:02:20.731312 1011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:20.731935 1011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:20.733106 1011 hybrid_clock.cc:648] HybridClock initialized: now 1761379340733086 us; error 45 us; skew 500 ppm
I20251025 08:02:20.734948 1011 webserver.cc:492] Webserver started at http://127.27.132.1:41641/ using document root <none> and password file <none>
I20251025 08:02:20.735589 1011 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:20.735654 1011 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:20.735853 1011 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:20.737634 1011 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "0f0c3af190cc40c58fa91559ba1f8c42"
format_stamp: "Formatted at 2025-10-25 08:02:20 on dist-test-slave-lq8l"
server_key: "526be2a7d8888c0b78fe2e40f757426b"
server_key_iv: "2f89e14d91539b991a791c0b698ea59b"
server_key_version: "encryptionkey@0"
I20251025 08:02:20.738137 1011 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "0f0c3af190cc40c58fa91559ba1f8c42"
format_stamp: "Formatted at 2025-10-25 08:02:20 on dist-test-slave-lq8l"
server_key: "526be2a7d8888c0b78fe2e40f757426b"
server_key_iv: "2f89e14d91539b991a791c0b698ea59b"
server_key_version: "encryptionkey@0"
I20251025 08:02:20.741672 1011 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:02:20.743952 1025 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:20.745004 1011 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 08:02:20.745106 1011 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "0f0c3af190cc40c58fa91559ba1f8c42"
format_stamp: "Formatted at 2025-10-25 08:02:20 on dist-test-slave-lq8l"
server_key: "526be2a7d8888c0b78fe2e40f757426b"
server_key_iv: "2f89e14d91539b991a791c0b698ea59b"
server_key_version: "encryptionkey@0"
I20251025 08:02:20.745211 1011 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:20.750211 1011 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:20.750837 1011 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:20.751004 1011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:20.751582 1011 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:20.752545 1011 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:20.752593 1011 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:20.752633 1011 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:20.752666 1011 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:20.761759 1011 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:46145
I20251025 08:02:20.761782 1138 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:46145 every 8 connection(s)
I20251025 08:02:20.762749 1011 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:02:20.768239 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.764576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:50539 (local address 127.27.132.62:45065)
1025 08:02:20.764739 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:20.764744 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:20.765710 (+ 966us) server_negotiation.cc:408] Connection header received
1025 08:02:20.766486 (+ 776us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:20.766490 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:20.766534 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:20.766626 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:20.767840 (+ 1214us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:20.767844 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:20.767846 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:20.767858 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:20.767880 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:20.767887 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:20.767978 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:20.768088 (+ 110us) server_negotiation.cc:300] Negotiation successful
1025 08:02:20.768126 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251025 08:02:20.769009 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.764892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:45065 (local address 127.27.132.1:50539)
1025 08:02:20.765563 (+ 671us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:20.765598 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:20.766287 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:20.766777 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:20.766785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:20.766876 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:20.767580 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:02:20.767582 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:20.767618 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:02:20.767676 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:20.767966 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:20.767971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:20.767975 (+ 4us) client_negotiation.cc:769] Sending connection context
1025 08:02:20.768138 (+ 163us) client_negotiation.cc:240] Negotiation successful
1025 08:02:20.768331 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20251025 08:02:20.770475 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 1011
I20251025 08:02:20.770498 1139 heartbeater.cc:344] Connected to a master server at 127.27.132.62:45065
I20251025 08:02:20.770586 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:02:20.770745 1139 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:20.770843 28176 external_mini_cluster.cc:1468] Setting key 7841c88df2a2a62152d4046add7d6841
I20251025 08:02:20.771178 1139 heartbeater.cc:507] Master 127.27.132.62:45065 requested a full tablet report, sending...
I20251025 08:02:20.772686 946 ts_manager.cc:194] Registered new tserver with Master: 0f0c3af190cc40c58fa91559ba1f8c42 (127.27.132.1:46145)
I20251025 08:02:20.772723 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:45065
--builtin_ntp_servers=127.27.132.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251025 08:02:20.883584 1142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:20.883828 1142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:20.883886 1142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:20.887205 1142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:20.887276 1142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:20.887398 1142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:02:20.891300 1142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:20.892550 1142 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:20.893812 1142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:20.900704 1147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:20.900698 1148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:20.900700 1150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:20.900890 1142 server_base.cc:1047] running on GCE node
I20251025 08:02:20.901345 1142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:20.901927 1142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:20.903102 1142 hybrid_clock.cc:648] HybridClock initialized: now 1761379340903088 us; error 29 us; skew 500 ppm
I20251025 08:02:20.905217 1142 webserver.cc:492] Webserver started at http://127.27.132.2:45949/ using document root <none> and password file <none>
I20251025 08:02:20.905833 1142 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:20.905893 1142 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:20.906095 1142 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:20.907914 1142 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "4f609222b378436294da4717913bf920"
format_stamp: "Formatted at 2025-10-25 08:02:20 on dist-test-slave-lq8l"
server_key: "97292022f685e2785c2e72ed454ffb84"
server_key_iv: "7613025c4bb93014cdb97855d0790ee1"
server_key_version: "encryptionkey@0"
I20251025 08:02:20.908401 1142 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "4f609222b378436294da4717913bf920"
format_stamp: "Formatted at 2025-10-25 08:02:20 on dist-test-slave-lq8l"
server_key: "97292022f685e2785c2e72ed454ffb84"
server_key_iv: "7613025c4bb93014cdb97855d0790ee1"
server_key_version: "encryptionkey@0"
I20251025 08:02:20.911944 1142 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:02:20.914302 1156 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:20.915453 1142 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:20.915578 1142 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "4f609222b378436294da4717913bf920"
format_stamp: "Formatted at 2025-10-25 08:02:20 on dist-test-slave-lq8l"
server_key: "97292022f685e2785c2e72ed454ffb84"
server_key_iv: "7613025c4bb93014cdb97855d0790ee1"
server_key_version: "encryptionkey@0"
I20251025 08:02:20.915680 1142 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:20.920663 1142 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:20.921272 1142 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:20.921453 1142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:20.922013 1142 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:20.922991 1142 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:20.923043 1142 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:20.923105 1142 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:20.923141 1142 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:20.932822 1142 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:36131
I20251025 08:02:20.932854 1269 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:36131 every 8 connection(s)
I20251025 08:02:20.933863 1142 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:02:20.938638 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 1142
I20251025 08:02:20.938773 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:02:20.939015 28176 external_mini_cluster.cc:1468] Setting key bd030a08dcafc852760458c76f65d1ae
I20251025 08:02:20.939743 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.935768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:38073 (local address 127.27.132.62:45065)
1025 08:02:20.935970 (+ 202us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:20.935974 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:20.936954 (+ 980us) server_negotiation.cc:408] Connection header received
1025 08:02:20.937824 (+ 870us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:20.937828 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:20.937881 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:20.937966 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:20.939228 (+ 1262us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:20.939233 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:20.939235 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:20.939248 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:20.939269 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:20.939277 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:20.939390 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:20.939552 (+ 162us) server_negotiation.cc:300] Negotiation successful
1025 08:02:20.939603 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20251025 08:02:20.940474 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:20.936081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:45065 (local address 127.27.132.2:38073)
1025 08:02:20.936803 (+ 722us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:20.936841 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:20.937596 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:20.938148 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:20.938156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:20.938246 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:20.938931 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:02:20.938933 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:20.938968 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:02:20.939027 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:20.939380 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:20.939389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:20.939395 (+ 6us) client_negotiation.cc:769] Sending connection context
1025 08:02:20.939607 (+ 212us) client_negotiation.cc:240] Negotiation successful
1025 08:02:20.939848 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
I20251025 08:02:20.941262 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:45065
--builtin_ntp_servers=127.27.132.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251025 08:02:20.941954 1270 heartbeater.cc:344] Connected to a master server at 127.27.132.62:45065
I20251025 08:02:20.942224 1270 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:20.942745 1270 heartbeater.cc:507] Master 127.27.132.62:45065 requested a full tablet report, sending...
I20251025 08:02:20.943933 946 ts_manager.cc:194] Registered new tserver with Master: 4f609222b378436294da4717913bf920 (127.27.132.2:36131)
W20251025 08:02:21.086587 1273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:21.086858 1273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:21.086949 1273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:21.090394 1273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:21.090468 1273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:21.090590 1273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:02:21.094671 1273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:21.095966 1273 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:21.097137 1273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:21.104074 1278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:21.104089 1281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:21.104074 1279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:21.104408 1273 server_base.cc:1047] running on GCE node
I20251025 08:02:21.104852 1273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:21.105492 1273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:21.106678 1273 hybrid_clock.cc:648] HybridClock initialized: now 1761379341106663 us; error 43 us; skew 500 ppm
I20251025 08:02:21.108772 1273 webserver.cc:492] Webserver started at http://127.27.132.3:34725/ using document root <none> and password file <none>
I20251025 08:02:21.109401 1273 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:21.109472 1273 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:21.109774 1273 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:21.111675 1273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "37baa065fe98472cbbfbfce49d1abf13"
format_stamp: "Formatted at 2025-10-25 08:02:21 on dist-test-slave-lq8l"
server_key: "71985d3b04c668d492ad729d4aaf89d9"
server_key_iv: "c975864fb81efbbad35f7cfedd22ee0c"
server_key_version: "encryptionkey@0"
I20251025 08:02:21.112176 1273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "37baa065fe98472cbbfbfce49d1abf13"
format_stamp: "Formatted at 2025-10-25 08:02:21 on dist-test-slave-lq8l"
server_key: "71985d3b04c668d492ad729d4aaf89d9"
server_key_iv: "c975864fb81efbbad35f7cfedd22ee0c"
server_key_version: "encryptionkey@0"
I20251025 08:02:21.115844 1273 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251025 08:02:21.118487 1287 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:21.119714 1273 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:21.119860 1273 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "37baa065fe98472cbbfbfce49d1abf13"
format_stamp: "Formatted at 2025-10-25 08:02:21 on dist-test-slave-lq8l"
server_key: "71985d3b04c668d492ad729d4aaf89d9"
server_key_iv: "c975864fb81efbbad35f7cfedd22ee0c"
server_key_version: "encryptionkey@0"
I20251025 08:02:21.119969 1273 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:21.124877 1273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:21.125476 1273 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:21.125656 1273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:21.126174 1273 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:21.127175 1273 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:21.127228 1273 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:21.127288 1273 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:21.127314 1273 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:21.136636 1273 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:42133
I20251025 08:02:21.136657 1400 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:42133 every 8 connection(s)
I20251025 08:02:21.137674 1273 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:02:21.138391 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 1273
I20251025 08:02:21.138509 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:02:21.138911 28176 external_mini_cluster.cc:1468] Setting key 5bb277112eec42feb88758b76085a3f3
I20251025 08:02:21.149497 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.139488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:44321 (local address 127.27.132.62:45065)
1025 08:02:21.139677 (+ 189us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:21.139682 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:21.141715 (+ 2033us) server_negotiation.cc:408] Connection header received
1025 08:02:21.142765 (+ 1050us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:21.142773 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:21.142826 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:21.142932 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:21.148802 (+ 5870us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:21.148808 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:21.148811 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:21.148835 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:21.148868 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:21.148880 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:21.149018 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:21.149264 (+ 246us) server_negotiation.cc:300] Negotiation successful
1025 08:02:21.149325 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20251025 08:02:21.150370 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.139978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:45065 (local address 127.27.132.3:44321)
1025 08:02:21.141562 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:21.141610 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:21.142533 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:21.143131 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:21.143143 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:21.143260 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:21.148378 (+ 5118us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:02:21.148383 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:21.148453 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:02:21.148565 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:21.149031 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:21.149043 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:21.149050 (+ 7us) client_negotiation.cc:769] Sending connection context
1025 08:02:21.149324 (+ 274us) client_negotiation.cc:240] Negotiation successful
1025 08:02:21.149631 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":145,"threads_started":1}
I20251025 08:02:21.151816 1401 heartbeater.cc:344] Connected to a master server at 127.27.132.62:45065
I20251025 08:02:21.152115 1401 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:21.152653 1401 heartbeater.cc:507] Master 127.27.132.62:45065 requested a full tablet report, sending...
I20251025 08:02:21.153751 946 ts_manager.cc:194] Registered new tserver with Master: 37baa065fe98472cbbfbfce49d1abf13 (127.27.132.3:42133)
I20251025 08:02:21.165311 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 08:02:21.169027 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.167239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.27.132.62:45065)
1025 08:02:21.167445 (+ 206us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:21.167452 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:21.167538 (+ 86us) server_negotiation.cc:408] Connection header received
1025 08:02:21.167672 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:21.167675 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:21.167728 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:21.167800 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:21.168589 (+ 789us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:21.168592 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:21.168595 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:21.168607 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:21.168629 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:21.168637 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:21.168725 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:21.168803 (+ 78us) server_negotiation.cc:300] Negotiation successful
1025 08:02:21.168847 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20251025 08:02:21.171743 946 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:02:21.174175 946 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:02:21.185905 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.182494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:36131 (local address 127.0.0.1:39160)
1025 08:02:21.182913 (+ 419us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:21.182958 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:21.183073 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:21.184219 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:21.184225 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:21.184300 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:21.185171 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:02:21.185174 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:21.185186 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:02:21.185211 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:21.185566 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:21.185569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:21.185572 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 08:02:21.185638 (+ 66us) client_negotiation.cc:240] Negotiation successful
1025 08:02:21.185707 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":102,"threads_started":1}
I20251025 08:02:21.185911 1414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.183141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:42133 (local address 127.0.0.1:45130)
1025 08:02:21.183461 (+ 320us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:21.183476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:21.183546 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:21.184361 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:21.184365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:21.184380 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:21.185102 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:02:21.185107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:21.185147 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:02:21.185210 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:21.185522 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:21.185533 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:21.185541 (+ 8us) client_negotiation.cc:769] Sending connection context
1025 08:02:21.185645 (+ 104us) client_negotiation.cc:240] Negotiation successful
1025 08:02:21.185705 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":112,"threads_started":1}
I20251025 08:02:21.186491 1416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.183300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.27.132.3:42133)
1025 08:02:21.183955 (+ 655us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:21.183963 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:21.183987 (+ 24us) server_negotiation.cc:408] Connection header received
1025 08:02:21.184050 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:21.184056 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:21.184210 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:21.184594 (+ 384us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:21.185350 (+ 756us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:21.185354 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:21.185362 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:21.185372 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:21.185417 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:21.185427 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:21.185856 (+ 429us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:21.185902 (+ 46us) server_negotiation.cc:300] Negotiation successful
1025 08:02:21.185975 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":156,"threads_started":1}
I20251025 08:02:21.186663 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.182653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.27.132.2:36131)
1025 08:02:21.183569 (+ 916us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:21.183578 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:21.183607 (+ 29us) server_negotiation.cc:408] Connection header received
1025 08:02:21.183671 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:21.183679 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:21.183899 (+ 220us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:21.184049 (+ 150us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:21.185343 (+ 1294us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:21.185349 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:21.185360 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:21.185374 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:21.185434 (+ 60us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:21.185450 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:21.186355 (+ 905us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:21.186402 (+ 47us) server_negotiation.cc:300] Negotiation successful
1025 08:02:21.186469 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":136,"threads_started":1}
I20251025 08:02:21.187175 1415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.183505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:46145 (local address 127.0.0.1:48866)
1025 08:02:21.184212 (+ 707us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:21.184226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:21.184296 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:21.185332 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:21.185335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:21.185348 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:21.186158 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:02:21.186160 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:21.186170 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:02:21.186194 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:21.186936 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:21.186939 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:21.186942 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 08:02:21.187025 (+ 83us) client_negotiation.cc:240] Negotiation successful
1025 08:02:21.187060 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":97,"threads_started":1}
I20251025 08:02:21.187417 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.183925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.27.132.1:46145)
1025 08:02:21.184728 (+ 803us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:21.184735 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:21.184757 (+ 22us) server_negotiation.cc:408] Connection header received
1025 08:02:21.184846 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:21.184853 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:21.184998 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:21.185537 (+ 539us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:21.186405 (+ 868us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:21.186411 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:21.186420 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:21.186443 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:21.186487 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:21.186499 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:21.186953 (+ 454us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:21.187034 (+ 81us) server_negotiation.cc:300] Negotiation successful
1025 08:02:21.187094 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":104,"threads_started":1}
I20251025 08:02:21.189534 1204 tablet_service.cc:1505] Processing CreateTablet for tablet 50e8ebe17a7348c694300c162a2f7922 (DEFAULT_TABLE table=test-table [id=49909ed9b123439f806abca7a999397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:21.189535 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 50e8ebe17a7348c694300c162a2f7922 (DEFAULT_TABLE table=test-table [id=49909ed9b123439f806abca7a999397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:21.189687 1073 tablet_service.cc:1505] Processing CreateTablet for tablet 50e8ebe17a7348c694300c162a2f7922 (DEFAULT_TABLE table=test-table [id=49909ed9b123439f806abca7a999397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:21.190606 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e8ebe17a7348c694300c162a2f7922. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:21.190606 1204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e8ebe17a7348c694300c162a2f7922. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:21.190608 1073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e8ebe17a7348c694300c162a2f7922. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:21.196264 1418 tablet_bootstrap.cc:492] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42: Bootstrap starting.
I20251025 08:02:21.197677 1420 tablet_bootstrap.cc:492] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920: Bootstrap starting.
I20251025 08:02:21.198697 1418 tablet_bootstrap.cc:654] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:21.199605 1418 log.cc:826] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:21.199637 1419 tablet_bootstrap.cc:492] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: Bootstrap starting.
I20251025 08:02:21.199851 1420 tablet_bootstrap.cc:654] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:21.200693 1420 log.cc:826] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:21.201740 1419 tablet_bootstrap.cc:654] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:21.202106 1418 tablet_bootstrap.cc:492] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42: No bootstrap required, opened a new log
I20251025 08:02:21.202302 1418 ts_tablet_manager.cc:1403] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251025 08:02:21.202549 1419 log.cc:826] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:21.203110 1420 tablet_bootstrap.cc:492] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920: No bootstrap required, opened a new log
I20251025 08:02:21.203294 1420 ts_tablet_manager.cc:1403] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251025 08:02:21.204492 1419 tablet_bootstrap.cc:492] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: No bootstrap required, opened a new log
I20251025 08:02:21.204676 1419 ts_tablet_manager.cc:1403] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 08:02:21.204974 1418 raft_consensus.cc:359] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f609222b378436294da4717913bf920" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 36131 } } peers { permanent_uuid: "37baa065fe98472cbbfbfce49d1abf13" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42133 } } peers { permanent_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46145 } }
I20251025 08:02:21.205170 1418 raft_consensus.cc:385] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:21.205227 1418 raft_consensus.cc:740] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0c3af190cc40c58fa91559ba1f8c42, State: Initialized, Role: FOLLOWER
I20251025 08:02:21.205647 1418 consensus_queue.cc:260] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f609222b378436294da4717913bf920" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 36131 } } peers { permanent_uuid: "37baa065fe98472cbbfbfce49d1abf13" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42133 } } peers { permanent_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46145 } }
I20251025 08:02:21.206573 1418 ts_tablet_manager.cc:1434] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251025 08:02:21.206481 1420 raft_consensus.cc:359] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f609222b378436294da4717913bf920" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 36131 } } peers { permanent_uuid: "37baa065fe98472cbbfbfce49d1abf13" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42133 } } peers { permanent_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46145 } }
I20251025 08:02:21.206678 1420 raft_consensus.cc:385] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:21.206729 1420 raft_consensus.cc:740] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f609222b378436294da4717913bf920, State: Initialized, Role: FOLLOWER
I20251025 08:02:21.206947 1139 heartbeater.cc:499] Master 127.27.132.62:45065 was elected leader, sending a full tablet report...
I20251025 08:02:21.207182 1420 consensus_queue.cc:260] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f609222b378436294da4717913bf920" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 36131 } } peers { permanent_uuid: "37baa065fe98472cbbfbfce49d1abf13" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42133 } } peers { permanent_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46145 } }
I20251025 08:02:21.207798 1419 raft_consensus.cc:359] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f609222b378436294da4717913bf920" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 36131 } } peers { permanent_uuid: "37baa065fe98472cbbfbfce49d1abf13" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42133 } } peers { permanent_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46145 } }
I20251025 08:02:21.207998 1419 raft_consensus.cc:385] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:21.208019 1270 heartbeater.cc:499] Master 127.27.132.62:45065 was elected leader, sending a full tablet report...
I20251025 08:02:21.208045 1419 raft_consensus.cc:740] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37baa065fe98472cbbfbfce49d1abf13, State: Initialized, Role: FOLLOWER
I20251025 08:02:21.208135 1420 ts_tablet_manager.cc:1434] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20251025 08:02:21.208462 1419 consensus_queue.cc:260] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f609222b378436294da4717913bf920" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 36131 } } peers { permanent_uuid: "37baa065fe98472cbbfbfce49d1abf13" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42133 } } peers { permanent_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46145 } }
I20251025 08:02:21.209244 1419 ts_tablet_manager.cc:1434] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251025 08:02:21.209648 1401 heartbeater.cc:499] Master 127.27.132.62:45065 was elected leader, sending a full tablet report...
W20251025 08:02:21.263929 1140 tablet.cc:2378] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 08:02:21.324697 1425 raft_consensus.cc:493] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:02:21.324931 1425 raft_consensus.cc:515] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f609222b378436294da4717913bf920" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 36131 } } peers { permanent_uuid: "37baa065fe98472cbbfbfce49d1abf13" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42133 } } peers { permanent_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46145 } }
I20251025 08:02:21.326133 1425 leader_election.cc:290] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37baa065fe98472cbbfbfce49d1abf13 (127.27.132.3:42133), 0f0c3af190cc40c58fa91559ba1f8c42 (127.27.132.1:46145)
I20251025 08:02:21.328207 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.326399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:42133 (local address 127.27.132.2:45497)
1025 08:02:21.326546 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:21.326563 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:21.326656 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:21.326962 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:21.326967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:21.326985 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:21.327590 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:02:21.327593 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:21.327611 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:02:21.327641 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:21.327960 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:21.327964 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:21.327967 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 08:02:21.328060 (+ 93us) client_negotiation.cc:240] Negotiation successful
1025 08:02:21.328106 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20251025 08:02:21.328315 1416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.326488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:45497 (local address 127.27.132.3:42133)
1025 08:02:21.326626 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:21.326634 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:21.326656 (+ 22us) server_negotiation.cc:408] Connection header received
1025 08:02:21.326778 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:21.326781 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:21.326829 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:21.326914 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:21.327782 (+ 868us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:21.327788 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:21.327791 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:21.327804 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:21.327823 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:21.327831 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:21.327911 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:21.328100 (+ 189us) server_negotiation.cc:300] Negotiation successful
1025 08:02:21.328152 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251025 08:02:21.328944 1355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e8ebe17a7348c694300c162a2f7922" candidate_uuid: "4f609222b378436294da4717913bf920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37baa065fe98472cbbfbfce49d1abf13" is_pre_election: true
I20251025 08:02:21.329182 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.326535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:53069 (local address 127.27.132.1:46145)
1025 08:02:21.326661 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:21.326665 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:21.327054 (+ 389us) server_negotiation.cc:408] Connection header received
1025 08:02:21.327142 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:21.327145 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:21.327195 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:21.327287 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:21.328788 (+ 1501us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:21.328791 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:21.328793 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:21.328804 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:21.328823 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:21.328830 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:21.328914 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:21.329022 (+ 108us) server_negotiation.cc:300] Negotiation successful
1025 08:02:21.329061 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251025 08:02:21.329217 1427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.326406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:46145 (local address 127.27.132.2:53069)
1025 08:02:21.326951 (+ 545us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:21.326965 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:21.327056 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:21.327351 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:21.327358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:21.327375 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:21.328578 (+ 1203us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:02:21.328581 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:21.328593 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:02:21.328618 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:21.328920 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:21.328925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:21.328931 (+ 6us) client_negotiation.cc:769] Sending connection context
1025 08:02:21.329013 (+ 82us) client_negotiation.cc:240] Negotiation successful
1025 08:02:21.329051 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":150,"thread_start_us":171,"threads_started":1}
I20251025 08:02:21.329222 1355 raft_consensus.cc:2468] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f609222b378436294da4717913bf920 in term 0.
I20251025 08:02:21.329730 1158 leader_election.cc:304] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37baa065fe98472cbbfbfce49d1abf13, 4f609222b378436294da4717913bf920; no voters:
I20251025 08:02:21.329869 1093 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e8ebe17a7348c694300c162a2f7922" candidate_uuid: "4f609222b378436294da4717913bf920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" is_pre_election: true
I20251025 08:02:21.330003 1425 raft_consensus.cc:2804] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:02:21.330085 1425 raft_consensus.cc:493] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:02:21.330142 1425 raft_consensus.cc:3060] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:21.330145 1093 raft_consensus.cc:2468] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f609222b378436294da4717913bf920 in term 0.
I20251025 08:02:21.331210 1425 raft_consensus.cc:515] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f609222b378436294da4717913bf920" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 36131 } } peers { permanent_uuid: "37baa065fe98472cbbfbfce49d1abf13" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42133 } } peers { permanent_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46145 } }
I20251025 08:02:21.331605 1425 leader_election.cc:290] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [CANDIDATE]: Term 1 election: Requested vote from peers 37baa065fe98472cbbfbfce49d1abf13 (127.27.132.3:42133), 0f0c3af190cc40c58fa91559ba1f8c42 (127.27.132.1:46145)
I20251025 08:02:21.331959 1355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e8ebe17a7348c694300c162a2f7922" candidate_uuid: "4f609222b378436294da4717913bf920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37baa065fe98472cbbfbfce49d1abf13"
I20251025 08:02:21.332005 1093 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e8ebe17a7348c694300c162a2f7922" candidate_uuid: "4f609222b378436294da4717913bf920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0c3af190cc40c58fa91559ba1f8c42"
I20251025 08:02:21.332082 1355 raft_consensus.cc:3060] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:21.332113 1093 raft_consensus.cc:3060] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:21.333056 1093 raft_consensus.cc:2468] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f609222b378436294da4717913bf920 in term 1.
I20251025 08:02:21.333386 1160 leader_election.cc:304] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0c3af190cc40c58fa91559ba1f8c42, 4f609222b378436294da4717913bf920; no voters:
I20251025 08:02:21.333539 1355 raft_consensus.cc:2468] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f609222b378436294da4717913bf920 in term 1.
I20251025 08:02:21.333602 1425 raft_consensus.cc:2804] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:21.333876 1425 raft_consensus.cc:697] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 1 LEADER]: Becoming Leader. State: Replica: 4f609222b378436294da4717913bf920, State: Running, Role: LEADER
I20251025 08:02:21.334203 1425 consensus_queue.cc:237] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f609222b378436294da4717913bf920" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 36131 } } peers { permanent_uuid: "37baa065fe98472cbbfbfce49d1abf13" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42133 } } peers { permanent_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46145 } }
I20251025 08:02:21.337566 946 catalog_manager.cc:5649] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f609222b378436294da4717913bf920 (127.27.132.2). New cstate: current_term: 1 leader_uuid: "4f609222b378436294da4717913bf920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f609222b378436294da4717913bf920" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 36131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37baa065fe98472cbbfbfce49d1abf13" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46145 } health_report { overall_health: UNKNOWN } } }
I20251025 08:02:21.378547 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.376954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39170 (local address 127.27.132.2:36131)
1025 08:02:21.377108 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:21.377116 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:21.377133 (+ 17us) server_negotiation.cc:408] Connection header received
1025 08:02:21.377221 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:21.377223 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:21.377284 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:21.377361 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:21.378147 (+ 786us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:21.378150 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:21.378152 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:21.378161 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:21.378179 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:21.378186 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:21.378265 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:21.378349 (+ 84us) server_negotiation.cc:300] Negotiation successful
1025 08:02:21.378390 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251025 08:02:21.384479 1355 raft_consensus.cc:1275] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13 [term 1 FOLLOWER]: Refusing update from remote peer 4f609222b378436294da4717913bf920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:21.384491 1093 raft_consensus.cc:1275] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42 [term 1 FOLLOWER]: Refusing update from remote peer 4f609222b378436294da4717913bf920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:21.385262 1425 consensus_queue.cc:1048] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37baa065fe98472cbbfbfce49d1abf13" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:21.385457 1428 consensus_queue.cc:1048] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0c3af190cc40c58fa91559ba1f8c42" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 08:02:21.388589 1402 tablet.cc:2378] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 08:02:21.393787 1432 mvcc.cc:204] Tried to move back new op lower bound from 7214609782302752768 to 7214609782108020736. Current Snapshot: MvccSnapshot[applied={T|T < 7214609782302752768}]
I20251025 08:02:21.395067 1433 mvcc.cc:204] Tried to move back new op lower bound from 7214609782302752768 to 7214609782108020736. Current Snapshot: MvccSnapshot[applied={T|T < 7214609782302752768}]
I20251025 08:02:21.401686 946 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39850:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:21.401849 946 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39850:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:21.404435 946 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b6e434f9ddf6476da5662cd0b3e3c17e: Sending DeleteTablet for 3 replicas of tablet 50e8ebe17a7348c694300c162a2f7922
I20251025 08:02:21.405323 1073 tablet_service.cc:1552] Processing DeleteTablet for tablet 50e8ebe17a7348c694300c162a2f7922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:21 UTC) from {username='slave'} at 127.0.0.1:48866
I20251025 08:02:21.405354 1204 tablet_service.cc:1552] Processing DeleteTablet for tablet 50e8ebe17a7348c694300c162a2f7922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:21 UTC) from {username='slave'} at 127.0.0.1:39160
I20251025 08:02:21.405872 1442 tablet_replica.cc:333] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920: stopping tablet replica
I20251025 08:02:21.405869 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 50e8ebe17a7348c694300c162a2f7922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:21 UTC) from {username='slave'} at 127.0.0.1:45130
I20251025 08:02:21.406143 1442 raft_consensus.cc:2243] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:02:21.406550 1441 tablet_replica.cc:333] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42: stopping tablet replica
I20251025 08:02:21.406674 1443 tablet_replica.cc:333] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: stopping tablet replica
I20251025 08:02:21.406816 1441 raft_consensus.cc:2243] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:21.407025 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 1011
I20251025 08:02:21.407178 1441 raft_consensus.cc:2272] T 50e8ebe17a7348c694300c162a2f7922 P 0f0c3af190cc40c58fa91559ba1f8c42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:21.407025 1443 raft_consensus.cc:2243] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:21.408121 1443 raft_consensus.cc:2272] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:21.409715 1443 ts_tablet_manager.cc:1916] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:21.411453 1442 raft_consensus.cc:2272] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:21.412182 1443 ts_tablet_manager.cc:1929] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:21.412289 1443 log.cc:1199] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/50e8ebe17a7348c694300c162a2f7922
I20251025 08:02:21.412631 1443 ts_tablet_manager.cc:1950] T 50e8ebe17a7348c694300c162a2f7922 P 37baa065fe98472cbbfbfce49d1abf13: Deleting consensus metadata
I20251025 08:02:21.413322 1442 ts_tablet_manager.cc:1916] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:21.413947 936 catalog_manager.cc:4985] TS 37baa065fe98472cbbfbfce49d1abf13 (127.27.132.3:42133): tablet 50e8ebe17a7348c694300c162a2f7922 (table test-table [id=49909ed9b123439f806abca7a999397a]) successfully deleted
W20251025 08:02:21.416034 938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.132.1:46145 (error 108)
I20251025 08:02:21.416082 1442 ts_tablet_manager.cc:1929] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:21.416162 1442 log.cc:1199] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/50e8ebe17a7348c694300c162a2f7922
I20251025 08:02:21.416527 1442 ts_tablet_manager.cc:1950] T 50e8ebe17a7348c694300c162a2f7922 P 4f609222b378436294da4717913bf920: Deleting consensus metadata
I20251025 08:02:21.417246 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 1142
I20251025 08:02:21.417279 1415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.416948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:46145 (local address 127.0.0.1:48878)
1025 08:02:21.417103 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:21.417178 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.1:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20251025 08:02:21.417443 938 catalog_manager.cc:4712] TS 0f0c3af190cc40c58fa91559ba1f8c42 (127.27.132.1:46145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50e8ebe17a7348c694300c162a2f7922: Network error: Client connection negotiation failed: client connection to 127.27.132.1:46145: connect: Connection refused (error 111)
I20251025 08:02:21.422873 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 1273
I20251025 08:02:21.424166 1415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.423122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:36131 (local address 127.0.0.1:39172)
1025 08:02:21.424032 (+ 910us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:21.424083 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.2:36131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":820}
I20251025 08:02:21.429237 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 920
2025-10-25T08:02:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1072 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20251025 08:02:21.451897 28176 mini_oidc.cc:150] Starting JWKS server
I20251025 08:02:21.453313 28176 webserver.cc:492] Webserver started at https://127.0.0.1:39853/ using document root <none> and password file <none>
I20251025 08:02:21.453441 28176 mini_oidc.cc:186] Starting OIDC Discovery server
I20251025 08:02:21.453641 28176 webserver.cc:492] Webserver started at http://127.0.0.1:40379/ using document root <none> and password file <none>
2025-10-25T08:02:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:21Z Disabled control of system clock
I20251025 08:02:21.461964 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:35055
--webserver_interface=127.27.132.62
--webserver_port=0
--builtin_ntp_servers=127.27.132.20:35885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:35055
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39853/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/testchainca.cert with env {}
W20251025 08:02:21.572386 1450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:21.572651 1450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:21.572743 1450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:21.576079 1450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:21.576160 1450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:21.576186 1450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:21.576220 1450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:21.576258 1450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251025 08:02:21.576301 1450 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251025 08:02:21.576333 1450 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39853/jwks.json
I20251025 08:02:21.580927 1450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:35885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:35055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:35055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39853/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:21.582100 1450 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:21.582980 1450 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:21.588726 1455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:21.588781 1458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:21.588860 1450 server_base.cc:1047] running on GCE node
W20251025 08:02:21.588726 1456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:21.589514 1450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:21.590433 1450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:21.591589 1450 hybrid_clock.cc:648] HybridClock initialized: now 1761379341591584 us; error 29 us; skew 500 ppm
I20251025 08:02:21.593389 1450 webserver.cc:492] Webserver started at http://127.27.132.62:39415/ using document root <none> and password file <none>
I20251025 08:02:21.593972 1450 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:21.594039 1450 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:21.594228 1450 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:21.596040 1450 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "dfa3e5542c32441294fa0dc8e3a61a1f"
format_stamp: "Formatted at 2025-10-25 08:02:21 on dist-test-slave-lq8l"
server_key: "1a777bcab12fd89dbd89de1ea598cd3c"
server_key_iv: "b1f9f08b4914003c2edadba06575b78b"
server_key_version: "encryptionkey@0"
I20251025 08:02:21.596524 1450 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "dfa3e5542c32441294fa0dc8e3a61a1f"
format_stamp: "Formatted at 2025-10-25 08:02:21 on dist-test-slave-lq8l"
server_key: "1a777bcab12fd89dbd89de1ea598cd3c"
server_key_iv: "b1f9f08b4914003c2edadba06575b78b"
server_key_version: "encryptionkey@0"
I20251025 08:02:21.599902 1450 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:02:21.602245 1464 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:21.603286 1450 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:21.603480 1450 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "dfa3e5542c32441294fa0dc8e3a61a1f"
format_stamp: "Formatted at 2025-10-25 08:02:21 on dist-test-slave-lq8l"
server_key: "1a777bcab12fd89dbd89de1ea598cd3c"
server_key_iv: "b1f9f08b4914003c2edadba06575b78b"
server_key_version: "encryptionkey@0"
I20251025 08:02:21.603582 1450 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:21.629464 1450 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:21.630205 1450 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:21.630419 1450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:21.638298 1450 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:35055
I20251025 08:02:21.638304 1516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:35055 every 8 connection(s)
I20251025 08:02:21.639441 1450 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:21.642309 1517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:21.647980 1517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f: Bootstrap starting.
I20251025 08:02:21.648012 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 1450
I20251025 08:02:21.648123 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:21.648383 28176 external_mini_cluster.cc:1468] Setting key 305d51e09b05f2b797a3f4348fb2e716
I20251025 08:02:21.651182 1517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:21.651950 1517 log.cc:826] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:21.653865 1517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f: No bootstrap required, opened a new log
I20251025 08:02:21.656576 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.649665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.27.132.62:35055)
1025 08:02:21.650290 (+ 625us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:21.650300 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:21.650335 (+ 35us) server_negotiation.cc:408] Connection header received
1025 08:02:21.650920 (+ 585us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:21.650938 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:21.651317 (+ 379us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:21.651686 (+ 369us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:21.652235 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:21.652998 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:21.653821 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:21.654160 (+ 339us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:21.654767 (+ 607us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:21.654788 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:21.654801 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:21.655190 (+ 389us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:21.655234 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:21.655249 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:21.655384 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:21.655705 (+ 321us) server_negotiation.cc:300] Negotiation successful
1025 08:02:21.655952 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
I20251025 08:02:21.656872 1517 raft_consensus.cc:359] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa3e5542c32441294fa0dc8e3a61a1f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35055 } }
I20251025 08:02:21.657083 1517 raft_consensus.cc:385] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:21.657153 1517 raft_consensus.cc:740] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa3e5542c32441294fa0dc8e3a61a1f, State: Initialized, Role: FOLLOWER
I20251025 08:02:21.657657 1517 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa3e5542c32441294fa0dc8e3a61a1f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35055 } }
I20251025 08:02:21.657825 1517 raft_consensus.cc:399] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:21.657891 1517 raft_consensus.cc:493] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:21.657984 1517 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:21.659090 1517 raft_consensus.cc:515] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa3e5542c32441294fa0dc8e3a61a1f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35055 } }
I20251025 08:02:21.659518 1517 leader_election.cc:304] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfa3e5542c32441294fa0dc8e3a61a1f; no voters:
I20251025 08:02:21.659858 28176 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251025 08:02:21.659874 1517 leader_election.cc:290] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:21.659973 1522 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:21.660147 1522 raft_consensus.cc:697] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [term 1 LEADER]: Becoming Leader. State: Replica: dfa3e5542c32441294fa0dc8e3a61a1f, State: Running, Role: LEADER
I20251025 08:02:21.660452 1522 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa3e5542c32441294fa0dc8e3a61a1f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35055 } }
I20251025 08:02:21.661119 1517 sys_catalog.cc:565] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:21.662143 1523 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfa3e5542c32441294fa0dc8e3a61a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa3e5542c32441294fa0dc8e3a61a1f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35055 } } }
I20251025 08:02:21.662272 1523 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:21.662602 1524 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfa3e5542c32441294fa0dc8e3a61a1f. Latest consensus state: current_term: 1 leader_uuid: "dfa3e5542c32441294fa0dc8e3a61a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa3e5542c32441294fa0dc8e3a61a1f" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 35055 } } }
I20251025 08:02:21.662664 1531 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:21.662775 1524 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:21.665592 1531 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:21.671445 1531 catalog_manager.cc:1357] Generated new cluster ID: 2e586c9b7ceb435ab5881b36eb336905
I20251025 08:02:21.671538 1531 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:21.684027 1531 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:21.685010 1531 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:21.698696 1531 catalog_manager.cc:6022] T 00000000000000000000000000000000 P dfa3e5542c32441294fa0dc8e3a61a1f: Generated new TSK 0
I20251025 08:02:21.699553 1531 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 08:02:21.741042 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.716094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.27.132.62:35055)
1025 08:02:21.716244 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:21.716249 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:21.716354 (+ 105us) server_negotiation.cc:408] Connection header received
1025 08:02:21.716595 (+ 241us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:21.716599 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:21.716671 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:21.716798 (+ 127us) server_negotiation.cc:227] Negotiated authn=JWT
1025 08:02:21.717200 (+ 402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:21.718071 (+ 871us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:21.718816 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:21.719042 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:21.719164 (+ 122us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 08:02:21.740497 (+ 21333us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
1025 08:02:21.740619 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:21.740821 (+ 202us) server_negotiation.cc:300] Negotiation successful
1025 08:02:21.740889 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"thread_start_us":166,"threads_started":1}
I20251025 08:02:21.752063 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:21.748314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.27.132.62:35055)
1025 08:02:21.748454 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:21.748459 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:21.748607 (+ 148us) server_negotiation.cc:408] Connection header received
1025 08:02:21.748751 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:21.748754 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:21.748802 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:21.748869 (+ 67us) server_negotiation.cc:227] Negotiated authn=JWT
1025 08:02:21.749218 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:21.749747 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:21.750389 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:21.750723 (+ 334us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:21.750781 (+ 58us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 08:02:21.751695 (+ 914us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1025 08:02:21.751929 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42426: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":56}
W20251025 08:02:21.752157 1520 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42426: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20251025 08:02:21.752835 1553 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:21.748196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:35055 (local address 127.0.0.1:42426)
1025 08:02:21.748511 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:21.748530 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:21.748646 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:21.748889 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:21.748892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:21.748899 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
1025 08:02:21.749104 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:21.749110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:21.749861 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:21.749865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:21.750259 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:21.750265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:21.750402 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:21.750428 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1025 08:02:21.751901 (+ 1473us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1025 08:02:21.752062 (+ 161us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.132.62:35055: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":112,"threads_started":1}
I20251025 08:02:36.761651 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:36.758321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51148 (local address 127.27.132.62:35055)
1025 08:02:36.758598 (+ 277us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:36.758602 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:36.758634 (+ 32us) server_negotiation.cc:408] Connection header received
1025 08:02:36.758686 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:36.758689 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:36.758735 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:36.758817 (+ 82us) server_negotiation.cc:227] Negotiated authn=JWT
1025 08:02:36.759194 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:36.759798 (+ 604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:36.760545 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:36.760708 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:36.760756 (+ 48us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 08:02:36.761379 (+ 623us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1025 08:02:36.761532 (+ 153us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51148: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
W20251025 08:02:36.761721 1559 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51148: JWT verification failed: token expired
W20251025 08:02:36.761788 1558 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:36.758115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:35055 (local address 127.0.0.1:51148)
1025 08:02:36.758439 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:36.758454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:36.758558 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:36.758838 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:36.758840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:36.758845 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
1025 08:02:36.759047 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:36.759053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:36.759962 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:36.759966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:36.760397 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:36.760407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:36.760536 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:36.760559 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1025 08:02:36.761524 (+ 965us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1025 08:02:36.761650 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.132.62:35055: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":120,"threads_started":1}
W20251025 08:02:36.764535 1559 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:36.763643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.27.132.62:35055)
1025 08:02:36.763796 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:36.763800 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:36.763885 (+ 85us) server_negotiation.cc:408] Connection header received
1025 08:02:36.763975 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:36.763977 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:36.764026 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:36.764098 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:36.764372 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51156: BlockingRecv error: recv got EOF from 127.0.0.1:51156 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20251025 08:02:36.768577 1570 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.27.132.62:35055
W20251025 08:02:36.769301 1559 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:36.768102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.27.132.62:35055)
1025 08:02:36.768288 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:36.768293 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:36.768493 (+ 200us) server_negotiation.cc:408] Connection header received
1025 08:02:36.768833 (+ 340us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:36.768837 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:36.768894 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:36.768981 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:36.769189 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51164: BlockingRecv error: recv got EOF from 127.0.0.1:51164 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20251025 08:02:36.769671 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 1450
2025-10-25T08:02:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15359 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20251025 08:02:36.810989 28176 mini_oidc.cc:150] Starting JWKS server
I20251025 08:02:36.812004 28176 webserver.cc:492] Webserver started at https://127.0.0.1:40529/ using document root <none> and password file <none>
I20251025 08:02:36.812090 28176 mini_oidc.cc:186] Starting OIDC Discovery server
I20251025 08:02:36.812306 28176 webserver.cc:492] Webserver started at http://127.0.0.1:36317/ using document root <none> and password file <none>
2025-10-25T08:02:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:36Z Disabled control of system clock
I20251025 08:02:36.820901 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:40493
--webserver_interface=127.27.132.62
--webserver_port=0
--builtin_ntp_servers=127.27.132.20:37119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:40493
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40529/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/testchainca.cert with env {}
W20251025 08:02:36.928885 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:36.929157 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:36.929208 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:36.932808 1576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:36.932883 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:36.932900 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:36.932920 1576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:36.932937 1576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251025 08:02:36.932961 1576 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251025 08:02:36.932976 1576 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40529/jwks.json
I20251025 08:02:36.937559 1576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:40493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40529/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:36.938668 1576 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:36.939529 1576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:36.945554 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:36.945583 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:36.945755 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:36.945878 1576 server_base.cc:1047] running on GCE node
I20251025 08:02:36.946362 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:36.947283 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:36.948494 1576 hybrid_clock.cc:648] HybridClock initialized: now 1761379356948471 us; error 41 us; skew 500 ppm
I20251025 08:02:36.950488 1576 webserver.cc:492] Webserver started at http://127.27.132.62:46529/ using document root <none> and password file <none>
I20251025 08:02:36.951054 1576 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:36.951112 1576 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:36.951294 1576 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:36.953006 1576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "b4e5349414a8415a9e10c2eb8707d02d"
format_stamp: "Formatted at 2025-10-25 08:02:36 on dist-test-slave-lq8l"
server_key: "1401579c941d1ab65426ca0fe260f2a8"
server_key_iv: "4e86e7490a1d20bff9744e620d0b5194"
server_key_version: "encryptionkey@0"
I20251025 08:02:36.953482 1576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "b4e5349414a8415a9e10c2eb8707d02d"
format_stamp: "Formatted at 2025-10-25 08:02:36 on dist-test-slave-lq8l"
server_key: "1401579c941d1ab65426ca0fe260f2a8"
server_key_iv: "4e86e7490a1d20bff9744e620d0b5194"
server_key_version: "encryptionkey@0"
I20251025 08:02:36.956892 1576 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20251025 08:02:36.959129 1590 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:36.960155 1576 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20251025 08:02:36.960266 1576 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "b4e5349414a8415a9e10c2eb8707d02d"
format_stamp: "Formatted at 2025-10-25 08:02:36 on dist-test-slave-lq8l"
server_key: "1401579c941d1ab65426ca0fe260f2a8"
server_key_iv: "4e86e7490a1d20bff9744e620d0b5194"
server_key_version: "encryptionkey@0"
I20251025 08:02:36.960361 1576 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:36.980026 1576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:36.980720 1576 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:36.980904 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:36.988413 1576 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:40493
I20251025 08:02:36.988438 1642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:40493 every 8 connection(s)
I20251025 08:02:36.989481 1576 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:36.992381 1643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:36.996209 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 1576
I20251025 08:02:36.996345 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:36.996575 28176 external_mini_cluster.cc:1468] Setting key 3e2b7db6be37309c7e0ce025c84ad882
I20251025 08:02:36.998164 1643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d: Bootstrap starting.
I20251025 08:02:37.000494 1643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:37.001197 1643 log.cc:826] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:37.003141 1643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d: No bootstrap required, opened a new log
I20251025 08:02:37.004304 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:36.997850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.27.132.62:40493)
1025 08:02:36.998557 (+ 707us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:36.998567 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:36.998603 (+ 36us) server_negotiation.cc:408] Connection header received
1025 08:02:36.999179 (+ 576us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:36.999209 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:36.999618 (+ 409us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:36.999936 (+ 318us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:37.000454 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.001226 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:37.001918 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.002218 (+ 300us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:37.002733 (+ 515us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:37.002752 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:37.002762 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:37.003133 (+ 371us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:37.003174 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:37.003188 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:37.003288 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:37.003484 (+ 196us) server_negotiation.cc:300] Negotiation successful
1025 08:02:37.003715 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
I20251025 08:02:37.005841 1643 raft_consensus.cc:359] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5349414a8415a9e10c2eb8707d02d" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40493 } }
I20251025 08:02:37.006058 1643 raft_consensus.cc:385] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:37.006120 1643 raft_consensus.cc:740] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e5349414a8415a9e10c2eb8707d02d, State: Initialized, Role: FOLLOWER
I20251025 08:02:37.006589 1643 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5349414a8415a9e10c2eb8707d02d" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40493 } }
I20251025 08:02:37.006749 1643 raft_consensus.cc:399] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:37.006812 1643 raft_consensus.cc:493] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:37.006904 1643 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:37.007097 28176 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251025 08:02:37.007896 1643 raft_consensus.cc:515] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5349414a8415a9e10c2eb8707d02d" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40493 } }
I20251025 08:02:37.008275 1643 leader_election.cc:304] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4e5349414a8415a9e10c2eb8707d02d; no voters:
I20251025 08:02:37.008638 1643 leader_election.cc:290] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:37.008751 1648 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:37.009002 1648 raft_consensus.cc:697] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [term 1 LEADER]: Becoming Leader. State: Replica: b4e5349414a8415a9e10c2eb8707d02d, State: Running, Role: LEADER
I20251025 08:02:37.009292 1648 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5349414a8415a9e10c2eb8707d02d" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40493 } }
I20251025 08:02:37.009771 1643 sys_catalog.cc:565] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:37.010929 1650 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4e5349414a8415a9e10c2eb8707d02d. Latest consensus state: current_term: 1 leader_uuid: "b4e5349414a8415a9e10c2eb8707d02d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5349414a8415a9e10c2eb8707d02d" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40493 } } }
I20251025 08:02:37.011091 1650 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:37.010928 1649 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4e5349414a8415a9e10c2eb8707d02d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5349414a8415a9e10c2eb8707d02d" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40493 } } }
I20251025 08:02:37.011307 1649 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:37.011747 1657 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:37.014433 1657 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:37.019922 1657 catalog_manager.cc:1357] Generated new cluster ID: dc64b03f51ba4796a0fbd70cbc77694d
I20251025 08:02:37.019999 1657 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:37.043622 1657 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:37.044778 1657 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:37.052127 1657 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b4e5349414a8415a9e10c2eb8707d02d: Generated new TSK 0
I20251025 08:02:37.052891 1657 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251025 08:02:37.083187 1673 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251025 08:02:37.083269 1673 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251025 08:02:37.084092 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:37.067747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.27.132.62:40493)
1025 08:02:37.067890 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:37.067895 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:37.068001 (+ 106us) server_negotiation.cc:408] Connection header received
1025 08:02:37.068164 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:37.068168 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:37.068233 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:37.068349 (+ 116us) server_negotiation.cc:227] Negotiated authn=JWT
1025 08:02:37.068742 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.069572 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:37.070242 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.070466 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:37.070582 (+ 116us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 08:02:37.083648 (+ 13066us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1025 08:02:37.083951 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20251025 08:02:37.084177 1646 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251025 08:02:37.084252 1672 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:37.067635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:40493 (local address 127.0.0.1:48860)
1025 08:02:37.067918 (+ 283us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:37.067931 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:37.068030 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:37.068391 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:37.068394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:37.068402 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1025 08:02:37.068628 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:37.068635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.069706 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:37.069709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:37.070141 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:37.070150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.070249 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:37.070270 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1025 08:02:37.083906 (+ 13636us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1025 08:02:37.084063 (+ 157us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.132.62:40493: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":104,"threads_started":1}
I20251025 08:02:37.085001 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 1576
2025-10-25T08:02:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (314 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20251025 08:02:37.124977 28176 mini_oidc.cc:150] Starting JWKS server
I20251025 08:02:37.125882 28176 webserver.cc:492] Webserver started at https://127.0.0.1:43475/ using document root <none> and password file <none>
I20251025 08:02:37.125986 28176 mini_oidc.cc:186] Starting OIDC Discovery server
I20251025 08:02:37.126217 28176 webserver.cc:492] Webserver started at http://127.0.0.1:37501/ using document root <none> and password file <none>
2025-10-25T08:02:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:37Z Disabled control of system clock
I20251025 08:02:37.134109 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:42565
--webserver_interface=127.27.132.62
--webserver_port=0
--builtin_ntp_servers=127.27.132.20:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:42565
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43475/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251025 08:02:37.243435 1679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:37.243669 1679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:37.243718 1679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:37.246904 1679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:37.246970 1679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:37.246987 1679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:37.247007 1679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:37.247028 1679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251025 08:02:37.247064 1679 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251025 08:02:37.247090 1679 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43475/jwks.json
I20251025 08:02:37.250869 1679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:42565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:42565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43475/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:37.251971 1679 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:37.253087 1679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:37.258711 1685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:37.258724 1684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:37.258711 1687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:37.258970 1679 server_base.cc:1047] running on GCE node
I20251025 08:02:37.259570 1679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:37.260452 1679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:37.261606 1679 hybrid_clock.cc:648] HybridClock initialized: now 1761379357261592 us; error 31 us; skew 500 ppm
I20251025 08:02:37.263376 1679 webserver.cc:492] Webserver started at http://127.27.132.62:42735/ using document root <none> and password file <none>
I20251025 08:02:37.263959 1679 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:37.264019 1679 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:37.264220 1679 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:37.265956 1679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "3bd3dd5b1d6a44659a6e54ae9bff62db"
format_stamp: "Formatted at 2025-10-25 08:02:37 on dist-test-slave-lq8l"
server_key: "9e1a358d56edad04fb3a5b5117514d6f"
server_key_iv: "2dd4ef382323c930b0edc93206c96d02"
server_key_version: "encryptionkey@0"
I20251025 08:02:37.266408 1679 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "3bd3dd5b1d6a44659a6e54ae9bff62db"
format_stamp: "Formatted at 2025-10-25 08:02:37 on dist-test-slave-lq8l"
server_key: "9e1a358d56edad04fb3a5b5117514d6f"
server_key_iv: "2dd4ef382323c930b0edc93206c96d02"
server_key_version: "encryptionkey@0"
I20251025 08:02:37.269752 1679 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251025 08:02:37.271910 1693 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:37.272905 1679 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:37.273008 1679 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "3bd3dd5b1d6a44659a6e54ae9bff62db"
format_stamp: "Formatted at 2025-10-25 08:02:37 on dist-test-slave-lq8l"
server_key: "9e1a358d56edad04fb3a5b5117514d6f"
server_key_iv: "2dd4ef382323c930b0edc93206c96d02"
server_key_version: "encryptionkey@0"
I20251025 08:02:37.273102 1679 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:37.291656 1679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:37.292348 1679 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:37.292553 1679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:37.299629 1679 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:42565
I20251025 08:02:37.299638 1745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:42565 every 8 connection(s)
I20251025 08:02:37.300645 1679 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:37.303398 1746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:37.308863 1746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db: Bootstrap starting.
I20251025 08:02:37.309573 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 1679
I20251025 08:02:37.309691 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:37.309911 28176 external_mini_cluster.cc:1468] Setting key b4301fa77cc7872ed110717b3d7b6745
I20251025 08:02:37.311504 1746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:37.312196 1746 log.cc:826] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:37.313903 1746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db: No bootstrap required, opened a new log
I20251025 08:02:37.316532 1746 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd3dd5b1d6a44659a6e54ae9bff62db" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42565 } }
I20251025 08:02:37.316735 1746 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:37.316810 1746 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd3dd5b1d6a44659a6e54ae9bff62db, State: Initialized, Role: FOLLOWER
I20251025 08:02:37.317268 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:37.310914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.27.132.62:42565)
1025 08:02:37.311573 (+ 659us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:37.311582 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:37.311622 (+ 40us) server_negotiation.cc:408] Connection header received
1025 08:02:37.312173 (+ 551us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:37.312190 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:37.312579 (+ 389us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:37.312893 (+ 314us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:37.313385 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.314146 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:37.314807 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.315103 (+ 296us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:37.315648 (+ 545us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:37.315666 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:37.315675 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:37.316047 (+ 372us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:37.316084 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:37.316098 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:37.316195 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:37.316358 (+ 163us) server_negotiation.cc:300] Negotiation successful
1025 08:02:37.316602 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
I20251025 08:02:37.317250 1746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd3dd5b1d6a44659a6e54ae9bff62db" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42565 } }
I20251025 08:02:37.317381 1746 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:37.317469 1746 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:37.317561 1746 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:37.318375 1746 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd3dd5b1d6a44659a6e54ae9bff62db" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42565 } }
I20251025 08:02:37.318733 1746 leader_election.cc:304] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bd3dd5b1d6a44659a6e54ae9bff62db; no voters:
I20251025 08:02:37.319031 1746 leader_election.cc:290] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:37.319126 1751 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:37.319361 1751 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [term 1 LEADER]: Becoming Leader. State: Replica: 3bd3dd5b1d6a44659a6e54ae9bff62db, State: Running, Role: LEADER
I20251025 08:02:37.319681 1751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd3dd5b1d6a44659a6e54ae9bff62db" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42565 } }
I20251025 08:02:37.320052 28176 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251025 08:02:37.320184 1746 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:37.321488 1753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bd3dd5b1d6a44659a6e54ae9bff62db. Latest consensus state: current_term: 1 leader_uuid: "3bd3dd5b1d6a44659a6e54ae9bff62db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd3dd5b1d6a44659a6e54ae9bff62db" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42565 } } }
I20251025 08:02:37.321589 1753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:37.321887 1752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bd3dd5b1d6a44659a6e54ae9bff62db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd3dd5b1d6a44659a6e54ae9bff62db" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42565 } } }
I20251025 08:02:37.321972 1752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:37.322223 1761 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:37.325327 1761 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:37.330595 1761 catalog_manager.cc:1357] Generated new cluster ID: 88149eb55afa40519a2f44cbbbf2d918
I20251025 08:02:37.330654 1761 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:37.343253 1761 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:37.344197 1761 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:37.353283 1761 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3bd3dd5b1d6a44659a6e54ae9bff62db: Generated new TSK 0
I20251025 08:02:37.354177 1761 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251025 08:02:37.387482 1776 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251025 08:02:37.387563 1776 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251025 08:02:37.389135 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:37.370491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.27.132.62:42565)
1025 08:02:37.370635 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:37.370640 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:37.370738 (+ 98us) server_negotiation.cc:408] Connection header received
1025 08:02:37.370891 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:37.370894 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:37.370960 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:37.371081 (+ 121us) server_negotiation.cc:227] Negotiated authn=JWT
1025 08:02:37.371411 (+ 330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.372730 (+ 1319us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:37.373375 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.373575 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:37.373690 (+ 115us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 08:02:37.388624 (+ 14934us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1025 08:02:37.388991 (+ 367us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20251025 08:02:37.389169 1775 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:37.370377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42565 (local address 127.0.0.1:51700)
1025 08:02:37.370641 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:37.370655 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:37.370753 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:37.371074 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:37.371076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:37.371081 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
1025 08:02:37.371279 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:37.371285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.372864 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:37.372867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:37.373275 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:37.373283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.373393 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:37.373415 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1025 08:02:37.388866 (+ 15451us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1025 08:02:37.389011 (+ 145us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.132.62:42565: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
W20251025 08:02:37.389211 1749 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20251025 08:02:37.389845 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 1679
2025-10-25T08:02:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-10-25T08:02:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:37Z Disabled control of system clock
I20251025 08:02:37.437409 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:44315
--webserver_interface=127.27.132.62
--webserver_port=0
--builtin_ntp_servers=127.27.132.20:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:44315
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/testchainca.cert with env {}
W20251025 08:02:37.546408 1780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:37.546672 1780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:37.546728 1780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:37.550086 1780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:37.550160 1780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:37.550177 1780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:37.550199 1780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:37.550217 1780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251025 08:02:37.550241 1780 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251025 08:02:37.550274 1780 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20251025 08:02:37.555116 1780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:44315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:44315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:37.556437 1780 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:37.557307 1780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:37.563014 1786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:37.563099 1780 server_base.cc:1047] running on GCE node
W20251025 08:02:37.563014 1785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:37.563014 1788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:37.563817 1780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:37.564795 1780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:37.565964 1780 hybrid_clock.cc:648] HybridClock initialized: now 1761379357565952 us; error 28 us; skew 500 ppm
I20251025 08:02:37.567855 1780 webserver.cc:492] Webserver started at http://127.27.132.62:46187/ using document root <none> and password file <none>
I20251025 08:02:37.568425 1780 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:37.568498 1780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:37.568686 1780 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:37.570505 1780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "c5193897ce6c46f8bc5080406563f4f1"
format_stamp: "Formatted at 2025-10-25 08:02:37 on dist-test-slave-lq8l"
server_key: "9e903d158a4568abbd12ed3f9743be2c"
server_key_iv: "11d091cd5de766e29604517d918d0435"
server_key_version: "encryptionkey@0"
I20251025 08:02:37.570998 1780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "c5193897ce6c46f8bc5080406563f4f1"
format_stamp: "Formatted at 2025-10-25 08:02:37 on dist-test-slave-lq8l"
server_key: "9e903d158a4568abbd12ed3f9743be2c"
server_key_iv: "11d091cd5de766e29604517d918d0435"
server_key_version: "encryptionkey@0"
I20251025 08:02:37.574617 1780 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251025 08:02:37.577080 1794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:37.578142 1780 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:37.578270 1780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "c5193897ce6c46f8bc5080406563f4f1"
format_stamp: "Formatted at 2025-10-25 08:02:37 on dist-test-slave-lq8l"
server_key: "9e903d158a4568abbd12ed3f9743be2c"
server_key_iv: "11d091cd5de766e29604517d918d0435"
server_key_version: "encryptionkey@0"
I20251025 08:02:37.578372 1780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:37.605866 1780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:37.606603 1780 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:37.606810 1780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:37.614403 1780 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:44315
I20251025 08:02:37.614408 1846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:44315 every 8 connection(s)
I20251025 08:02:37.615505 1780 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:37.618384 1847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:37.623458 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 1780
I20251025 08:02:37.623575 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:37.623800 28176 external_mini_cluster.cc:1468] Setting key b4ba173fa06f42819738c715bd699406
I20251025 08:02:37.624197 1847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1: Bootstrap starting.
I20251025 08:02:37.626564 1847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:37.627434 1847 log.cc:826] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:37.629470 1847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1: No bootstrap required, opened a new log
I20251025 08:02:37.631780 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:37.625043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.27.132.62:44315)
1025 08:02:37.625682 (+ 639us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:37.625692 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:37.625727 (+ 35us) server_negotiation.cc:408] Connection header received
1025 08:02:37.626273 (+ 546us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:37.626290 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:37.626668 (+ 378us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:37.626985 (+ 317us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:37.627663 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.628432 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:37.629315 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.629581 (+ 266us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:37.630138 (+ 557us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:37.630158 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:37.630168 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:02:37.630573 (+ 405us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:37.630615 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:02:37.630630 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:37.630736 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:37.630901 (+ 165us) server_negotiation.cc:300] Negotiation successful
1025 08:02:37.631121 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":127,"threads_started":1}
I20251025 08:02:37.632248 1847 raft_consensus.cc:359] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5193897ce6c46f8bc5080406563f4f1" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44315 } }
I20251025 08:02:37.632436 1847 raft_consensus.cc:385] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:37.632472 1847 raft_consensus.cc:740] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5193897ce6c46f8bc5080406563f4f1, State: Initialized, Role: FOLLOWER
I20251025 08:02:37.632887 1847 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5193897ce6c46f8bc5080406563f4f1" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44315 } }
I20251025 08:02:37.633016 1847 raft_consensus.cc:399] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:37.633085 1847 raft_consensus.cc:493] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:37.633179 1847 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:37.634159 1847 raft_consensus.cc:515] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5193897ce6c46f8bc5080406563f4f1" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44315 } }
I20251025 08:02:37.634521 1847 leader_election.cc:304] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5193897ce6c46f8bc5080406563f4f1; no voters:
I20251025 08:02:37.634553 28176 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251025 08:02:37.634821 1847 leader_election.cc:290] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:37.634912 1852 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:37.635079 1852 raft_consensus.cc:697] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [term 1 LEADER]: Becoming Leader. State: Replica: c5193897ce6c46f8bc5080406563f4f1, State: Running, Role: LEADER
I20251025 08:02:37.635401 1852 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5193897ce6c46f8bc5080406563f4f1" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44315 } }
I20251025 08:02:37.636037 1847 sys_catalog.cc:565] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:37.636962 1854 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5193897ce6c46f8bc5080406563f4f1. Latest consensus state: current_term: 1 leader_uuid: "c5193897ce6c46f8bc5080406563f4f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5193897ce6c46f8bc5080406563f4f1" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44315 } } }
I20251025 08:02:37.637476 1854 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:37.637362 1853 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5193897ce6c46f8bc5080406563f4f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5193897ce6c46f8bc5080406563f4f1" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 44315 } } }
I20251025 08:02:37.637992 1853 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1 [sys.catalog]: This master's current role is: LEADER
W20251025 08:02:37.641074 1867 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 08:02:37.641155 1867 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 08:02:37.641223 1862 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:37.641925 1862 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:37.647023 1862 catalog_manager.cc:1357] Generated new cluster ID: 60f5a72632014de29b931a823ec33227
I20251025 08:02:37.647119 1862 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:37.657647 1862 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:37.658623 1862 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:37.664357 1862 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c5193897ce6c46f8bc5080406563f4f1: Generated new TSK 0
I20251025 08:02:37.665185 1862 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 08:02:37.761775 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:37.680457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.27.132.62:44315)
1025 08:02:37.680616 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:37.680621 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:37.680694 (+ 73us) server_negotiation.cc:408] Connection header received
1025 08:02:37.680868 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:37.680873 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:37.680938 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:37.681054 (+ 116us) server_negotiation.cc:227] Negotiated authn=JWT
1025 08:02:37.681362 (+ 308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.682219 (+ 857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:37.682876 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.683093 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:37.683205 (+ 112us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1025 08:02:37.761255 (+ 78050us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1025 08:02:37.761608 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39740: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":74}
W20251025 08:02:37.761920 1876 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:37.680328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:44315 (local address 127.0.0.1:39740)
1025 08:02:37.680612 (+ 284us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:37.680625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:37.680729 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:37.681047 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:37.681049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:37.681054 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
1025 08:02:37.681257 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:37.681263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.682336 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:37.682343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:37.682772 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:37.682780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:37.682884 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:37.682908 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1025 08:02:37.761501 (+ 78593us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1025 08:02:37.761726 (+ 225us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.132.62:44315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
W20251025 08:02:37.762200 1850 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39740: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
I20251025 08:02:37.762588 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 1780
2025-10-25T08:02:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (352 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:02:38 dist-test-slave-lq8l krb5kdc[1888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:02:38 dist-test-slave-lq8l krb5kdc[1888](info): set up 2 sockets
Oct 25 08:02:38 dist-test-slave-lq8l krb5kdc[1888](info): commencing operation
krb5kdc: starting...
W20251025 08:02:39.996747 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.972s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:02:40 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:02:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:40.132351 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:34687
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:34687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:40.240165 1904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:40.240415 1904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:40.240473 1904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:40.243736 1904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:40.243798 1904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:40.243820 1904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:40.243839 1904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:40.243862 1904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:40.248085 1904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:34687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:34687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:40.249173 1904 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:40.250037 1904 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:40.255653 1909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:40.255719 1904 server_base.cc:1047] running on GCE node
W20251025 08:02:40.255672 1910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:40.255673 1912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:40.256304 1904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:40.257210 1904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:40.258385 1904 hybrid_clock.cc:648] HybridClock initialized: now 1761379360258355 us; error 42 us; skew 500 ppm
Oct 25 08:02:40 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379360, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:40.260934 1904 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:02:40.261957 1904 webserver.cc:492] Webserver started at http://127.27.132.62:33897/ using document root <none> and password file <none>
I20251025 08:02:40.262503 1904 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:40.262562 1904 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:40.262746 1904 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:40.264463 1904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "eafc9359d060446bbdbc4522d7a54093"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "90d59e09b222f18fb0e796ddcff8f1a5"
server_key_iv: "cb53bae97039e761e1b9f27f0e7d056a"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.264930 1904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "eafc9359d060446bbdbc4522d7a54093"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "90d59e09b222f18fb0e796ddcff8f1a5"
server_key_iv: "cb53bae97039e761e1b9f27f0e7d056a"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.268299 1904 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:02:40.270485 1919 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:40.271520 1904 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:40.271636 1904 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "eafc9359d060446bbdbc4522d7a54093"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "90d59e09b222f18fb0e796ddcff8f1a5"
server_key_iv: "cb53bae97039e761e1b9f27f0e7d056a"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.271724 1904 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:40.286170 1904 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:40.292290 1904 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:40.292503 1904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:40.299813 1904 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:34687
I20251025 08:02:40.299824 1971 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:34687 every 8 connection(s)
I20251025 08:02:40.300828 1904 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:40.303598 1972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:40.308250 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 1904
I20251025 08:02:40.308418 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:40.308672 28176 external_mini_cluster.cc:1468] Setting key baffb4239808dba59acdbcf7e5d2db8f
I20251025 08:02:40.309288 1972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093: Bootstrap starting.
I20251025 08:02:40.311712 1972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:40.312635 1972 log.cc:826] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:40.314522 1972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093: No bootstrap required, opened a new log
Oct 25 08:02:40 dist-test-slave-lq8l krb5kdc[1888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:40.317512 1972 raft_consensus.cc:359] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafc9359d060446bbdbc4522d7a54093" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34687 } }
I20251025 08:02:40.317723 1972 raft_consensus.cc:385] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:40.317782 1972 raft_consensus.cc:740] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafc9359d060446bbdbc4522d7a54093, State: Initialized, Role: FOLLOWER
I20251025 08:02:40.318280 1972 consensus_queue.cc:260] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafc9359d060446bbdbc4522d7a54093" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34687 } }
I20251025 08:02:40.318440 1972 raft_consensus.cc:399] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:40.318507 1972 raft_consensus.cc:493] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:40.318596 1972 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:40.319612 1972 raft_consensus.cc:515] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafc9359d060446bbdbc4522d7a54093" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34687 } }
I20251025 08:02:40.319969 1972 leader_election.cc:304] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eafc9359d060446bbdbc4522d7a54093; no voters:
I20251025 08:02:40.320302 1972 leader_election.cc:290] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:40.320406 1977 raft_consensus.cc:2804] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:40.320650 1977 raft_consensus.cc:697] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [term 1 LEADER]: Becoming Leader. State: Replica: eafc9359d060446bbdbc4522d7a54093, State: Running, Role: LEADER
I20251025 08:02:40.320951 1977 consensus_queue.cc:237] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafc9359d060446bbdbc4522d7a54093" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34687 } }
I20251025 08:02:40.321580 1972 sys_catalog.cc:565] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:40.322059 1977 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eafc9359d060446bbdbc4522d7a54093. Latest consensus state: current_term: 1 leader_uuid: "eafc9359d060446bbdbc4522d7a54093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafc9359d060446bbdbc4522d7a54093" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34687 } } }
I20251025 08:02:40.322175 1977 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:40.322670 1978 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eafc9359d060446bbdbc4522d7a54093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafc9359d060446bbdbc4522d7a54093" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34687 } } }
I20251025 08:02:40.322808 1978 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093 [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:40.323408 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:40.309887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.27.132.62:34687)
1025 08:02:40.310263 (+ 376us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:40.310272 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:40.310306 (+ 34us) server_negotiation.cc:408] Connection header received
1025 08:02:40.310831 (+ 525us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:40.310847 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:40.311128 (+ 281us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:40.311446 (+ 318us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:40.312494 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:40.313513 (+ 1019us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:40.314176 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:40.314452 (+ 276us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:40.316674 (+ 2222us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:40.316695 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:40.316709 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:40.316754 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:40.318961 (+ 2207us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:40.319409 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:40.319417 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:40.319424 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:40.319527 (+ 103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:40.319893 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:40.319897 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:40.319899 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:40.320344 (+ 445us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:40.321081 (+ 737us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:40.321562 (+ 481us) server_negotiation.cc:300] Negotiation successful
1025 08:02:40.321907 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
W20251025 08:02:40.326349 1993 catalog_manager.cc:1568] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 08:02:40.326423 1993 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 08:02:40.326488 1981 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:40.327160 1981 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:40.332108 1981 catalog_manager.cc:1357] Generated new cluster ID: 61922ee58cb041e391b0b5cf5867d826
I20251025 08:02:40.332168 1981 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:40.350025 1981 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:40.350984 1981 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:40.359164 1981 catalog_manager.cc:6022] T 00000000000000000000000000000000 P eafc9359d060446bbdbc4522d7a54093: Generated new TSK 0
I20251025 08:02:40.359926 1981 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:40.408396 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34687
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:40.515092 2000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:40.515357 2000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:40.515462 2000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:40.518689 2000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:40.518761 2000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:40.518868 2000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:02:40.523197 2000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:40.524353 2000 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:40.525219 2000 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:40.532171 2005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:40.532243 2000 server_base.cc:1047] running on GCE node
W20251025 08:02:40.532164 2008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:40.532142 2006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:40.532727 2000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:40.533331 2000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:40.534488 2000 hybrid_clock.cc:648] HybridClock initialized: now 1761379360534477 us; error 35 us; skew 500 ppm
Oct 25 08:02:40 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379360, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:40.537102 2000 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:02:40.538151 2000 webserver.cc:492] Webserver started at http://127.27.132.1:41821/ using document root <none> and password file <none>
I20251025 08:02:40.538708 2000 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:40.538764 2000 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:40.538944 2000 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:40.540653 2000 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "def9b7b7437f4a9387659401fa49c119"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "10d27a2a316e1b75c710223e9ab71f21"
server_key_iv: "0a2801b06f54480e66c2057e2a406b68"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.541117 2000 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "def9b7b7437f4a9387659401fa49c119"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "10d27a2a316e1b75c710223e9ab71f21"
server_key_iv: "0a2801b06f54480e66c2057e2a406b68"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.544575 2000 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251025 08:02:40.546788 2015 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:40.547921 2000 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:40.548034 2000 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "def9b7b7437f4a9387659401fa49c119"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "10d27a2a316e1b75c710223e9ab71f21"
server_key_iv: "0a2801b06f54480e66c2057e2a406b68"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.548122 2000 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:40.566743 2000 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:40.569667 2000 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:40.569876 2000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:40.570454 2000 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:40.571411 2000 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:40.571465 2000 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:40.571522 2000 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:40.571552 2000 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:40.580909 2000 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:35761
I20251025 08:02:40.580931 2128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:35761 every 8 connection(s)
I20251025 08:02:40.581897 2000 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:02:40.584152 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 2000
I20251025 08:02:40.584276 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:02:40.584555 28176 external_mini_cluster.cc:1468] Setting key 3af850001b44315fed3a0814b09d350b
Oct 25 08:02:40 dist-test-slave-lq8l krb5kdc[1888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379360, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
I20251025 08:02:40.596750 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:40.583671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:54107 (local address 127.27.132.62:34687)
1025 08:02:40.583822 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:40.583826 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:40.584520 (+ 694us) server_negotiation.cc:408] Connection header received
1025 08:02:40.585553 (+ 1033us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:40.585557 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:40.585610 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:40.585727 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:40.588029 (+ 2302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:40.588842 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:40.589787 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:40.589926 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:40.592947 (+ 3021us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:40.592968 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:40.592971 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:40.592998 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:40.594695 (+ 1697us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:40.595260 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:40.595263 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:40.595265 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:40.595314 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:40.595694 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:40.595700 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:40.595705 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:40.595893 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:40.595988 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:40.596490 (+ 502us) server_negotiation.cc:300] Negotiation successful
1025 08:02:40.596618 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
I20251025 08:02:40.597453 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:40.583899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34687 (local address 127.27.132.1:54107)
1025 08:02:40.584365 (+ 466us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:40.584410 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:40.585259 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:40.585910 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:40.585923 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:40.586477 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:40.587840 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:40.587860 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:40.588983 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:40.588989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:40.589658 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:40.589672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:40.589911 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:40.590789 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:40.590832 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:40.592745 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:40.594838 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:40.594848 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:40.594862 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:40.595144 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:40.595464 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:40.595466 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:40.595468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:40.595557 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:40.595995 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:40.596000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:40.596233 (+ 233us) client_negotiation.cc:769] Sending connection context
1025 08:02:40.596409 (+ 176us) client_negotiation.cc:240] Negotiation successful
1025 08:02:40.596641 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20251025 08:02:40.598670 2129 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34687
I20251025 08:02:40.598920 2129 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:40.599494 2129 heartbeater.cc:507] Master 127.27.132.62:34687 requested a full tablet report, sending...
I20251025 08:02:40.600960 1936 ts_manager.cc:194] Registered new tserver with Master: def9b7b7437f4a9387659401fa49c119 (127.27.132.1:35761)
I20251025 08:02:40.602126 1936 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:54107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:40.631803 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34687
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:40.736323 2136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:40.736557 2136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:40.736642 2136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:40.739887 2136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:40.739959 2136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:40.740063 2136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:02:40.744164 2136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:40.745198 2136 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:40.746032 2136 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:40.752414 2142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:40.752525 2136 server_base.cc:1047] running on GCE node
W20251025 08:02:40.752413 2144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:40.752419 2141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:40.753016 2136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:40.753549 2136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:40.754696 2136 hybrid_clock.cc:648] HybridClock initialized: now 1761379360754684 us; error 25 us; skew 500 ppm
Oct 25 08:02:40 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379360, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:40.757246 2136 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:02:40.758240 2136 webserver.cc:492] Webserver started at http://127.27.132.2:38053/ using document root <none> and password file <none>
I20251025 08:02:40.758790 2136 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:40.758845 2136 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:40.759028 2136 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:40.760730 2136 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "d7a9f495dd094691ad3a40cb816457ec"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "2dae205f98dab4e2dbf1e8f9dd66aaec"
server_key_iv: "7e6f33247e0d6c2668b12aa60a51832f"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.761188 2136 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "d7a9f495dd094691ad3a40cb816457ec"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "2dae205f98dab4e2dbf1e8f9dd66aaec"
server_key_iv: "7e6f33247e0d6c2668b12aa60a51832f"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.764423 2136 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20251025 08:02:40.766587 2151 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:40.767589 2136 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 08:02:40.767690 2136 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "d7a9f495dd094691ad3a40cb816457ec"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "2dae205f98dab4e2dbf1e8f9dd66aaec"
server_key_iv: "7e6f33247e0d6c2668b12aa60a51832f"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.767781 2136 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:40.782725 2136 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:40.785776 2136 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:40.785975 2136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:40.786545 2136 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:40.787424 2136 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:40.787480 2136 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:40.787540 2136 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:40.787575 2136 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:40.796463 2136 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:46653
I20251025 08:02:40.796489 2265 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:46653 every 8 connection(s)
I20251025 08:02:40.797384 2136 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
Oct 25 08:02:40 dist-test-slave-lq8l krb5kdc[1888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379360, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:40.807367 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 2136
I20251025 08:02:40.807510 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:02:40.807753 28176 external_mini_cluster.cc:1468] Setting key 07840a75b2f09ec8f1dbc2d3f74c80c6
I20251025 08:02:40.810334 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:40.799063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:46255 (local address 127.27.132.62:34687)
1025 08:02:40.799187 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:40.799191 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:40.799910 (+ 719us) server_negotiation.cc:408] Connection header received
1025 08:02:40.800723 (+ 813us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:40.800727 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:40.800789 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:40.800871 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:40.802687 (+ 1816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:40.803192 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:40.803891 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:40.804079 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:40.806557 (+ 2478us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:40.806582 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:40.806585 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:40.806620 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:40.808289 (+ 1669us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:40.808823 (+ 534us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:40.808826 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:40.808828 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:40.808870 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:40.809186 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:40.809189 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:40.809190 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:40.809377 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:40.809489 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:40.809959 (+ 470us) server_negotiation.cc:300] Negotiation successful
1025 08:02:40.810094 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251025 08:02:40.810832 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:40.799288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34687 (local address 127.27.132.2:46255)
1025 08:02:40.799775 (+ 487us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:40.799809 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:40.800526 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:40.801038 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:40.801046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:40.801441 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:40.802403 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:40.802420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:40.803338 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:40.803341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:40.803763 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:40.803771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:40.804001 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:40.804654 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:40.804678 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:40.806382 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:40.808398 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:40.808402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:40.808412 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:40.808734 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:40.808963 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:40.808966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:40.808970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:40.809084 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:40.809483 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:40.809488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:40.809717 (+ 229us) client_negotiation.cc:769] Sending connection context
1025 08:02:40.809885 (+ 168us) client_negotiation.cc:240] Negotiation successful
1025 08:02:40.810100 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":135,"threads_started":1}
I20251025 08:02:40.811946 2266 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34687
I20251025 08:02:40.812182 2266 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:40.812707 2266 heartbeater.cc:507] Master 127.27.132.62:34687 requested a full tablet report, sending...
I20251025 08:02:40.813704 1936 ts_manager.cc:194] Registered new tserver with Master: d7a9f495dd094691ad3a40cb816457ec (127.27.132.2:46653)
I20251025 08:02:40.814360 1936 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:46255
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:40.852306 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34687
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:40.958827 2273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:40.959071 2273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:40.959159 2273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:40.962314 2273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:40.962378 2273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:40.962492 2273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:02:40.966632 2273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:40.967720 2273 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:40.968546 2273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:40.975172 2279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:40.975270 2273 server_base.cc:1047] running on GCE node
W20251025 08:02:40.975227 2278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:40.975175 2281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:40.975838 2273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:40.976434 2273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:40.977602 2273 hybrid_clock.cc:648] HybridClock initialized: now 1761379360977585 us; error 32 us; skew 500 ppm
Oct 25 08:02:40 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379360, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:40.980316 2273 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:02:40.981338 2273 webserver.cc:492] Webserver started at http://127.27.132.3:43435/ using document root <none> and password file <none>
I20251025 08:02:40.981904 2273 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:40.981956 2273 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:40.982141 2273 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:40.983943 2273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "bf779d0a106545fb9ff1a62adcb67867"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "af7641ab00f66d0bc19f9aa3a4c876d1"
server_key_iv: "ca6cca33033149d6c0bd0a149f1f9665"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.984412 2273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "bf779d0a106545fb9ff1a62adcb67867"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "af7641ab00f66d0bc19f9aa3a4c876d1"
server_key_iv: "ca6cca33033149d6c0bd0a149f1f9665"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.988111 2273 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251025 08:02:40.991461 2288 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:40.992725 2273 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:02:40.992843 2273 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "bf779d0a106545fb9ff1a62adcb67867"
format_stamp: "Formatted at 2025-10-25 08:02:40 on dist-test-slave-lq8l"
server_key: "af7641ab00f66d0bc19f9aa3a4c876d1"
server_key_iv: "ca6cca33033149d6c0bd0a149f1f9665"
server_key_version: "encryptionkey@0"
I20251025 08:02:40.992954 2273 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:41.009338 2273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:41.012440 2273 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:41.012622 2273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:41.013161 2273 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:41.014021 2273 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:41.014068 2273 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:41.014110 2273 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:41.014139 2273 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:41.023082 2273 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:38337
I20251025 08:02:41.023097 2401 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:38337 every 8 connection(s)
I20251025 08:02:41.024147 2273 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:02:41.027925 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 2273
I20251025 08:02:41.028039 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:02:41.028307 28176 external_mini_cluster.cc:1468] Setting key 855c6b812adc4721ebb5b0898ee25cfb
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[1888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379360, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:41.038164 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:41.025869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:42917 (local address 127.27.132.62:34687)
1025 08:02:41.026029 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:41.026034 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:41.026778 (+ 744us) server_negotiation.cc:408] Connection header received
1025 08:02:41.027919 (+ 1141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:41.027923 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:41.027990 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:41.028095 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:41.030419 (+ 2324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:41.031234 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:41.032186 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:41.032336 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:41.034965 (+ 2629us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:41.034982 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:41.034984 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:41.035012 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:41.036543 (+ 1531us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:41.036990 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:41.036992 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:41.036993 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:41.037030 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:41.037233 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:41.037235 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:41.037237 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:41.037375 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:41.037447 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:41.037926 (+ 479us) server_negotiation.cc:300] Negotiation successful
1025 08:02:41.038040 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251025 08:02:41.038785 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:41.026157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34687 (local address 127.27.132.3:42917)
1025 08:02:41.026599 (+ 442us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:41.026648 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:41.027654 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:41.028296 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:41.028310 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:41.028935 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:41.030224 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:41.030244 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:41.031428 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:41.031432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:41.032060 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:41.032070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:41.032313 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:41.032971 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:41.033006 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:41.034791 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:41.036655 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:41.036660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:41.036670 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:41.036908 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:41.037093 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:41.037095 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:41.037097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:41.037173 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:41.037453 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:41.037458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:41.037678 (+ 220us) client_negotiation.cc:769] Sending connection context
1025 08:02:41.037864 (+ 186us) client_negotiation.cc:240] Negotiation successful
1025 08:02:41.038078 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":118,"threads_started":1}
I20251025 08:02:41.039873 2402 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34687
I20251025 08:02:41.040102 2402 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:41.040570 2402 heartbeater.cc:507] Master 127.27.132.62:34687 requested a full tablet report, sending...
I20251025 08:02:41.041554 1936 ts_manager.cc:194] Registered new tserver with Master: bf779d0a106545fb9ff1a62adcb67867 (127.27.132.3:38337)
I20251025 08:02:41.042104 1936 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:42917
I20251025 08:02:41.042119 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/test-admin.keytab.
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:41.057801 28176 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20251025 08:02:41.060495 2409 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:41.059099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.27.132.1:35761)
1025 08:02:41.059432 (+ 333us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:41.059439 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:41.059462 (+ 23us) server_negotiation.cc:408] Connection header received
1025 08:02:41.059512 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:41.059517 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:41.059699 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:41.059838 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:41.060327 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43166: BlockingRecv error: recv got EOF from 127.0.0.1:43166 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
W20251025 08:02:41.060596 28176 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5cc' of type FILE
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:41.061631 28176 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:41.062510 28176 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[1888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.1@KRBTEST.COM
I20251025 08:02:41.071501 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:41.063396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.27.132.1:35761)
1025 08:02:41.063543 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:41.063548 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:41.063563 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:02:41.063688 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:41.063693 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:41.063740 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:41.063810 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:41.064458 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:41.065187 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:41.065884 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:41.066137 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:41.068268 (+ 2131us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:41.068290 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:41.068301 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:41.068326 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:41.069920 (+ 1594us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:41.070307 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:41.070312 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:41.070313 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:41.070365 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:41.070599 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:41.070603 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:41.070605 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:41.070795 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:41.070986 (+ 191us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:41.071166 (+ 180us) server_negotiation.cc:300] Negotiation successful
1025 08:02:41.071310 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20251025 08:02:41.075096 2409 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:41.074153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.27.132.1:35761)
1025 08:02:41.074326 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:41.074329 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:41.074342 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:02:41.074479 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:41.074482 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:41.074528 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:41.074617 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:41.074998 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43184: BlockingRecv error: recv got EOF from 127.0.0.1:43184 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20251025 08:02:41.075227 28176 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5cc' of type FILE
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:41.076349 28176 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:41.077230 28176 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[1888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.1@KRBTEST.COM
I20251025 08:02:41.085062 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:41.078036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.27.132.1:35761)
1025 08:02:41.078171 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:41.078175 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:41.078189 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:02:41.078285 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:41.078288 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:41.078325 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:41.078393 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:41.078972 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:41.079600 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:41.080195 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:41.080336 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:41.082256 (+ 1920us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:41.082272 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:41.082275 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:41.082298 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:41.083728 (+ 1430us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:41.084084 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:41.084087 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:41.084088 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:41.084124 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:41.084348 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:41.084350 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:41.084351 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:41.084499 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:41.084601 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:41.084787 (+ 186us) server_negotiation.cc:300] Negotiation successful
1025 08:02:41.084872 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20251025 08:02:41.087746 2409 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:41.086771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.27.132.1:35761)
1025 08:02:41.086914 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:41.086920 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:41.086967 (+ 47us) server_negotiation.cc:408] Connection header received
1025 08:02:41.087097 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:41.087102 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:41.087141 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:41.087208 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:41.087633 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43206: BlockingRecv error: recv got EOF from 127.0.0.1:43206 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20251025 08:02:41.088028 28176 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761379300881738-28176-0/krb5kdc/krb5cc)
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:41.088999 28176 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[1888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:41.089836 28176 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[1888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.1@KRBTEST.COM
I20251025 08:02:41.097530 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:41.090641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.27.132.1:35761)
1025 08:02:41.090784 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:41.090787 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:41.090845 (+ 58us) server_negotiation.cc:408] Connection header received
1025 08:02:41.090936 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:41.090939 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:41.090973 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:41.091036 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:41.091620 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:41.092070 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:41.092609 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:41.092741 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:41.094699 (+ 1958us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:41.094716 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:41.094718 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:41.094741 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:41.096207 (+ 1466us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:41.096635 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:41.096638 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:41.096639 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:41.096692 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:41.096875 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:41.096878 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:41.096879 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:41.097011 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:41.097101 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:41.097253 (+ 152us) server_negotiation.cc:300] Negotiation successful
1025 08:02:41.097340 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20251025 08:02:41.098651 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 2000
I20251025 08:02:41.104725 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 2136
I20251025 08:02:41.110064 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 2273
I20251025 08:02:41.115489 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 1904
2025-10-25T08:02:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3112 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[2422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[2422](info): set up 2 sockets
Oct 25 08:02:41 dist-test-slave-lq8l krb5kdc[2422](info): commencing operation
krb5kdc: starting...
W20251025 08:02:43.116703 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.979s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:02:43 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:02:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
I20251025 08:02:43.257923 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:42629
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:43.369388 2438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:43.369647 2438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:43.369702 2438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:43.373092 2438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:43.373162 2438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:43.373186 2438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:43.373206 2438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:43.373224 2438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:43.377669 2438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:43.378793 2438 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:43.379658 2438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:43.385613 2444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:43.385613 2446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:43.385639 2443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:43.385802 2438 server_base.cc:1047] running on GCE node
I20251025 08:02:43.386423 2438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:43.387423 2438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:43.388576 2438 hybrid_clock.cc:648] HybridClock initialized: now 1761379363388557 us; error 37 us; skew 500 ppm
Oct 25 08:02:43 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379363, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:43.391455 2438 init.cc:377] Logged in from keytab as oryx/127.27.132.62@KRBTEST.COM (short username oryx)
I20251025 08:02:43.392549 2438 webserver.cc:492] Webserver started at http://127.27.132.62:41019/ using document root <none> and password file <none>
I20251025 08:02:43.393105 2438 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:43.393153 2438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:43.393389 2438 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:43.395083 2438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "d28b5159b5704014b011a9766e6f0e8c"
format_stamp: "Formatted at 2025-10-25 08:02:43 on dist-test-slave-lq8l"
server_key: "af0f1f0491c929abc6e743462ff94ab5"
server_key_iv: "34405531f0bcc71c8c6b5d3c7c8ac7a8"
server_key_version: "encryptionkey@0"
I20251025 08:02:43.395589 2438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "d28b5159b5704014b011a9766e6f0e8c"
format_stamp: "Formatted at 2025-10-25 08:02:43 on dist-test-slave-lq8l"
server_key: "af0f1f0491c929abc6e743462ff94ab5"
server_key_iv: "34405531f0bcc71c8c6b5d3c7c8ac7a8"
server_key_version: "encryptionkey@0"
I20251025 08:02:43.399017 2438 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20251025 08:02:43.401335 2453 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:43.402354 2438 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:43.402487 2438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "d28b5159b5704014b011a9766e6f0e8c"
format_stamp: "Formatted at 2025-10-25 08:02:43 on dist-test-slave-lq8l"
server_key: "af0f1f0491c929abc6e743462ff94ab5"
server_key_iv: "34405531f0bcc71c8c6b5d3c7c8ac7a8"
server_key_version: "encryptionkey@0"
I20251025 08:02:43.402583 2438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:43.440264 2438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:43.443377 2438 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:43.443571 2438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:43.450775 2438 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:42629
I20251025 08:02:43.450789 2505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:42629 every 8 connection(s)
I20251025 08:02:43.451817 2438 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:43.454337 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 2438
I20251025 08:02:43.454456 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:43.454681 2506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:43.454722 28176 external_mini_cluster.cc:1468] Setting key 8525352ebbe30381eccd696c05d3609f
I20251025 08:02:43.461539 2506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c: Bootstrap starting.
Oct 25 08:02:43 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:43.463927 2506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:43.464754 2506 log.cc:826] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:43.466512 2506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c: No bootstrap required, opened a new log
I20251025 08:02:43.467525 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:43.456028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.27.132.62:42629)
1025 08:02:43.456440 (+ 412us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:43.456449 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:43.456482 (+ 33us) server_negotiation.cc:408] Connection header received
1025 08:02:43.457069 (+ 587us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:43.457088 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:43.457374 (+ 286us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:43.457671 (+ 297us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:43.458750 (+ 1079us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:43.459612 (+ 862us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:43.460497 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:43.460711 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:43.463163 (+ 2452us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:43.463206 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:43.463216 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:43.463250 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:43.465115 (+ 1865us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:43.465532 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:43.465537 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:43.465541 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:43.465614 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:43.465834 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:43.465838 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:43.465840 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:43.466161 (+ 321us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:43.466354 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:43.466653 (+ 299us) server_negotiation.cc:300] Negotiation successful
1025 08:02:43.466908 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":125,"threads_started":1}
I20251025 08:02:43.469084 2506 raft_consensus.cc:359] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } }
I20251025 08:02:43.469280 2506 raft_consensus.cc:385] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:43.469338 2506 raft_consensus.cc:740] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d28b5159b5704014b011a9766e6f0e8c, State: Initialized, Role: FOLLOWER
I20251025 08:02:43.469741 2506 consensus_queue.cc:260] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } }
I20251025 08:02:43.469867 2506 raft_consensus.cc:399] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:43.469935 2506 raft_consensus.cc:493] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:43.470026 2506 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:43.470892 2506 raft_consensus.cc:515] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } }
I20251025 08:02:43.471212 2506 leader_election.cc:304] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d28b5159b5704014b011a9766e6f0e8c; no voters:
I20251025 08:02:43.471503 2506 leader_election.cc:290] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:43.471626 2511 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:43.471845 2511 raft_consensus.cc:697] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 1 LEADER]: Becoming Leader. State: Replica: d28b5159b5704014b011a9766e6f0e8c, State: Running, Role: LEADER
I20251025 08:02:43.472147 2511 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } }
I20251025 08:02:43.472606 2506 sys_catalog.cc:565] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:43.473624 2513 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28b5159b5704014b011a9766e6f0e8c. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } }
I20251025 08:02:43.473781 2513 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:43.474161 2520 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:43.474217 2512 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } }
I20251025 08:02:43.474320 2512 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:43.477888 2520 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:43.483470 2520 catalog_manager.cc:1357] Generated new cluster ID: 73f659919e404a6591f3f9e2aa29e7c7
I20251025 08:02:43.483559 2520 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:43.505151 2520 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:43.506204 2520 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:43.513274 2520 catalog_manager.cc:6022] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c: Generated new TSK 0
I20251025 08:02:43.513994 2520 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
I20251025 08:02:43.560405 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:42629
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:43.671489 2534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:43.671763 2534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:43.671826 2534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:43.675397 2534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:43.675469 2534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:43.675566 2534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:02:43.680015 2534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:43.681190 2534 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:43.682065 2534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:43.688864 2539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:43.688880 2540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:43.688993 2534 server_base.cc:1047] running on GCE node
W20251025 08:02:43.689152 2542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:43.689533 2534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:43.690115 2534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:43.691277 2534 hybrid_clock.cc:648] HybridClock initialized: now 1761379363691251 us; error 38 us; skew 500 ppm
Oct 25 08:02:43 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379363, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:43.696763 2534 init.cc:377] Logged in from keytab as oryx/127.27.132.1@KRBTEST.COM (short username oryx)
I20251025 08:02:43.697829 2534 webserver.cc:492] Webserver started at http://127.27.132.1:40489/ using document root <none> and password file <none>
I20251025 08:02:43.698371 2534 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:43.698421 2534 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:43.698577 2534 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:43.700299 2534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "9e8418b907064b84a8752c25c6fd5890"
format_stamp: "Formatted at 2025-10-25 08:02:43 on dist-test-slave-lq8l"
server_key: "162d083991e712b773d6f2bea60216f0"
server_key_iv: "ebcdb9cd1168d451ec7a74cbda39fe0d"
server_key_version: "encryptionkey@0"
I20251025 08:02:43.700742 2534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "9e8418b907064b84a8752c25c6fd5890"
format_stamp: "Formatted at 2025-10-25 08:02:43 on dist-test-slave-lq8l"
server_key: "162d083991e712b773d6f2bea60216f0"
server_key_iv: "ebcdb9cd1168d451ec7a74cbda39fe0d"
server_key_version: "encryptionkey@0"
I20251025 08:02:43.704149 2534 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251025 08:02:43.706328 2549 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:43.707403 2534 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 08:02:43.707515 2534 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "9e8418b907064b84a8752c25c6fd5890"
format_stamp: "Formatted at 2025-10-25 08:02:43 on dist-test-slave-lq8l"
server_key: "162d083991e712b773d6f2bea60216f0"
server_key_iv: "ebcdb9cd1168d451ec7a74cbda39fe0d"
server_key_version: "encryptionkey@0"
I20251025 08:02:43.707607 2534 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:43.721083 2534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:43.724833 2534 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:43.725056 2534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:43.725805 2534 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:43.726783 2534 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:43.726830 2534 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:43.726876 2534 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:43.726908 2534 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:43.735759 2534 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:46805
I20251025 08:02:43.735801 2662 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:46805 every 8 connection(s)
I20251025 08:02:43.736771 2534 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
Oct 25 08:02:43 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379363, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:43.746356 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 2534
I20251025 08:02:43.746467 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:02:43.746721 28176 external_mini_cluster.cc:1468] Setting key 3c072213bbcd389d59fcd8948c283cda
I20251025 08:02:43.749680 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:43.738585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:51601 (local address 127.27.132.62:42629)
1025 08:02:43.738725 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:43.738728 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:43.739388 (+ 660us) server_negotiation.cc:408] Connection header received
1025 08:02:43.740235 (+ 847us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:43.740239 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:43.740303 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:43.740383 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:43.741717 (+ 1334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:43.742229 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:43.742868 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:43.743063 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:43.745826 (+ 2763us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:43.745851 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:43.745853 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:43.745883 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:43.747488 (+ 1605us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:43.748016 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:43.748021 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:43.748023 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:43.748076 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:43.748410 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:43.748413 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:43.748414 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:43.748611 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:43.748711 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:43.749434 (+ 723us) server_negotiation.cc:300] Negotiation successful
1025 08:02:43.749559 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 08:02:43.750566 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:43.738789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.1:51601)
1025 08:02:43.739222 (+ 433us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:43.739259 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:43.740030 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:43.740560 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:43.740568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:43.740990 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:43.741555 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:43.741568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:43.742351 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:43.742354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:43.742752 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:43.742759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:43.742987 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:43.744138 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:43.744162 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:43.745635 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:43.747609 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:43.747613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:43.747624 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:43.747883 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:43.748179 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:43.748182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:43.748184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:43.748310 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:43.748730 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:43.748740 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:43.749100 (+ 360us) client_negotiation.cc:769] Sending connection context
1025 08:02:43.749378 (+ 278us) client_negotiation.cc:240] Negotiation successful
1025 08:02:43.749701 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20251025 08:02:43.751783 2663 heartbeater.cc:344] Connected to a master server at 127.27.132.62:42629
I20251025 08:02:43.752074 2663 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:43.752802 2663 heartbeater.cc:507] Master 127.27.132.62:42629 requested a full tablet report, sending...
I20251025 08:02:43.754739 2470 ts_manager.cc:194] Registered new tserver with Master: 9e8418b907064b84a8752c25c6fd5890 (127.27.132.1:46805)
I20251025 08:02:43.756460 2470 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:51601
WARNING: no policy specified for oryx/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
I20251025 08:02:43.792296 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:42629
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:43.898538 2670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:43.898789 2670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:43.898854 2670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:43.902184 2670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:43.902253 2670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:43.902357 2670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:02:43.906798 2670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:43.908133 2670 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:43.909153 2670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:43.917364 2676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:43.917353 2675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:43.917378 2678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:43.917737 2670 server_base.cc:1047] running on GCE node
I20251025 08:02:43.918254 2670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:43.919178 2670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:43.920485 2670 hybrid_clock.cc:648] HybridClock initialized: now 1761379363920470 us; error 49 us; skew 500 ppm
Oct 25 08:02:43 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379363, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:43.923996 2670 init.cc:377] Logged in from keytab as oryx/127.27.132.2@KRBTEST.COM (short username oryx)
I20251025 08:02:43.925035 2670 webserver.cc:492] Webserver started at http://127.27.132.2:33197/ using document root <none> and password file <none>
I20251025 08:02:43.925596 2670 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:43.925642 2670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:43.925807 2670 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:43.927544 2670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "c5f1b63c203848ba95feb28f30b6b891"
format_stamp: "Formatted at 2025-10-25 08:02:43 on dist-test-slave-lq8l"
server_key: "ced925c934a0f3bfe5c178a486814c3a"
server_key_iv: "994d565ff359ce2040e0e41d9b8ec497"
server_key_version: "encryptionkey@0"
I20251025 08:02:43.928082 2670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "c5f1b63c203848ba95feb28f30b6b891"
format_stamp: "Formatted at 2025-10-25 08:02:43 on dist-test-slave-lq8l"
server_key: "ced925c934a0f3bfe5c178a486814c3a"
server_key_iv: "994d565ff359ce2040e0e41d9b8ec497"
server_key_version: "encryptionkey@0"
I20251025 08:02:43.932937 2670 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251025 08:02:43.935814 2685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:43.937142 2670 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251025 08:02:43.937274 2670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "c5f1b63c203848ba95feb28f30b6b891"
format_stamp: "Formatted at 2025-10-25 08:02:43 on dist-test-slave-lq8l"
server_key: "ced925c934a0f3bfe5c178a486814c3a"
server_key_iv: "994d565ff359ce2040e0e41d9b8ec497"
server_key_version: "encryptionkey@0"
I20251025 08:02:43.937399 2670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:43.966962 2670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:43.970211 2670 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:43.970417 2670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:43.970959 2670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:43.971869 2670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:43.971922 2670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:43.971982 2670 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:43.972009 2670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:43.981074 2670 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:41457
I20251025 08:02:43.981110 2798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:41457 every 8 connection(s)
I20251025 08:02:43.982421 2670 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:02:43.984899 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 2670
I20251025 08:02:43.985019 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:02:43.985312 28176 external_mini_cluster.cc:1468] Setting key e4f30fe31e8ad995cfeb528eacab6610
Oct 25 08:02:43 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379363, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.132.3@KRBTEST.COM; defaulting to no policy
I20251025 08:02:43.996210 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:43.984226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:49999 (local address 127.27.132.62:42629)
1025 08:02:43.984399 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:43.984405 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:43.985408 (+ 1003us) server_negotiation.cc:408] Connection header received
1025 08:02:43.986386 (+ 978us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:43.986390 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:43.986438 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:43.986532 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:43.988278 (+ 1746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:43.988797 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:43.989428 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:43.989610 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:43.992807 (+ 3197us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:43.992826 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:43.992828 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:43.992853 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:43.994297 (+ 1444us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:43.994774 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:43.994778 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:43.994780 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:43.994826 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:43.995081 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:43.995084 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:43.995086 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:43.995277 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:43.995421 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:43.995912 (+ 491us) server_negotiation.cc:300] Negotiation successful
1025 08:02:43.996036 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.3@KRBTEST.COM" created.
I20251025 08:02:43.996977 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:43.984463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.2:49999)
1025 08:02:43.985222 (+ 759us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:43.985273 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:43.986169 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:43.986702 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:43.986718 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:43.987280 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:43.988110 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:43.988129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:43.988926 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:43.988929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:43.989316 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:43.989323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:43.989545 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:43.990778 (+ 1233us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:43.990800 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:43.992626 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:43.994408 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:43.994414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:43.994426 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:43.994672 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:43.994908 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:43.994913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:43.994915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:43.994995 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:43.995421 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:43.995429 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:43.995667 (+ 238us) client_negotiation.cc:769] Sending connection context
1025 08:02:43.995861 (+ 194us) client_negotiation.cc:240] Negotiation successful
1025 08:02:43.996131 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":103,"threads_started":1}
I20251025 08:02:43.998021 2799 heartbeater.cc:344] Connected to a master server at 127.27.132.62:42629
I20251025 08:02:43.998270 2799 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:43.998795 2799 heartbeater.cc:507] Master 127.27.132.62:42629 requested a full tablet report, sending...
I20251025 08:02:43.999854 2470 ts_manager.cc:194] Registered new tserver with Master: c5f1b63c203848ba95feb28f30b6b891 (127.27.132.2:41457)
I20251025 08:02:44.000398 2470 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:49999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
I20251025 08:02:44.030403 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:42629
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:44.137300 2806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:44.137540 2806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:44.137598 2806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:44.140900 2806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:44.140964 2806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:44.141042 2806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:02:44.145287 2806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:44.146287 2806 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:44.147099 2806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:44.153687 2811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:44.153687 2814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:44.153714 2812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:44.153839 2806 server_base.cc:1047] running on GCE node
I20251025 08:02:44.154361 2806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:44.154911 2806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:44.156070 2806 hybrid_clock.cc:648] HybridClock initialized: now 1761379364156058 us; error 34 us; skew 500 ppm
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379364, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:44.158946 2806 init.cc:377] Logged in from keytab as oryx/127.27.132.3@KRBTEST.COM (short username oryx)
I20251025 08:02:44.160044 2806 webserver.cc:492] Webserver started at http://127.27.132.3:45259/ using document root <none> and password file <none>
I20251025 08:02:44.160576 2806 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:44.160621 2806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:44.160776 2806 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:44.162420 2806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "384defb513fd42959725a27971ee3759"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "fb2dbbe5e5f56feea8a94e0a2622d16e"
server_key_iv: "0074d8b08c5815d1997182c40dc67a9f"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.162853 2806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "384defb513fd42959725a27971ee3759"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "fb2dbbe5e5f56feea8a94e0a2622d16e"
server_key_iv: "0074d8b08c5815d1997182c40dc67a9f"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.166133 2806 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:02:44.168677 2821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:44.169780 2806 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:44.169886 2806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "384defb513fd42959725a27971ee3759"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "fb2dbbe5e5f56feea8a94e0a2622d16e"
server_key_iv: "0074d8b08c5815d1997182c40dc67a9f"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.169978 2806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:44.190977 2806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:44.193915 2806 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:44.194070 2806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:44.194545 2806 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:44.195406 2806 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:44.195461 2806 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:44.195501 2806 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:44.195523 2806 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:44.204370 2806 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:42201
I20251025 08:02:44.204416 2935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:42201 every 8 connection(s)
I20251025 08:02:44.205292 2806 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:02:44.206058 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 2806
I20251025 08:02:44.206182 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:02:44.206446 28176 external_mini_cluster.cc:1468] Setting key d10791cfcfdf45c4828364200c08fb44
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379364, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:44.217906 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.207529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:34221 (local address 127.27.132.62:42629)
1025 08:02:44.207679 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.207683 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.208393 (+ 710us) server_negotiation.cc:408] Connection header received
1025 08:02:44.209255 (+ 862us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.209258 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.209304 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.209369 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.210677 (+ 1308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.211169 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.211930 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.212080 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.214710 (+ 2630us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.214726 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.214728 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.214753 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.216083 (+ 1330us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.216558 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.216562 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.216563 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.216604 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.216870 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.216873 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.216875 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.217026 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.217121 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.217606 (+ 485us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.217746 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251025 08:02:44.218503 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.207741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.3:34221)
1025 08:02:44.208241 (+ 500us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.208283 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.209078 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.209508 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.209518 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.209951 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.210518 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.210531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.211300 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.211304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.211823 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.211832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.212008 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.213062 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.213083 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.214550 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.216197 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.216202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.216222 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.216459 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.216700 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.216703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.216704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.216784 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.217132 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.217140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.217356 (+ 216us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.217556 (+ 200us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.217821 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":115,"threads_started":1}
I20251025 08:02:44.219620 2936 heartbeater.cc:344] Connected to a master server at 127.27.132.62:42629
I20251025 08:02:44.219861 2936 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:44.220393 2936 heartbeater.cc:507] Master 127.27.132.62:42629 requested a full tablet report, sending...
I20251025 08:02:44.221318 2470 ts_manager.cc:194] Registered new tserver with Master: 384defb513fd42959725a27971ee3759 (127.27.132.3:42201)
I20251025 08:02:44.221873 2470 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:34221
I20251025 08:02:44.232039 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.132.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.132.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.132.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.132.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
I20251025 08:02:44.274534 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.61:34389
--webserver_interface=127.27.132.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:42629,127.27.132.61:34389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:44.379381 2943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:44.379614 2943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:44.379683 2943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:44.382952 2943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:44.383018 2943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:44.383041 2943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:44.383059 2943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:44.383077 2943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:44.387496 2943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:42629,127.27.132.61:34389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.61:34389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.132.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
W20251025 08:02:44.387691 2943 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.132.62:42629,127.27.132.61:34389'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20251025 08:02:44.388584 2943 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:44.389411 2943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:44.394701 2948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:44.394716 2949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:44.394795 2951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:44.394987 2943 server_base.cc:1047] running on GCE node
I20251025 08:02:44.395448 2943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:44.396272 2943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:44.397428 2943 hybrid_clock.cc:648] HybridClock initialized: now 1761379364397402 us; error 42 us; skew 500 ppm
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379364, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:44.400182 2943 init.cc:377] Logged in from keytab as oryx/127.27.132.61@KRBTEST.COM (short username oryx)
I20251025 08:02:44.401225 2943 webserver.cc:492] Webserver started at http://127.27.132.61:41677/ using document root <none> and password file <none>
I20251025 08:02:44.401746 2943 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:44.401791 2943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:44.401947 2943 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:44.403602 2943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data/instance:
uuid: "7fb81dc334b641e084097ce2feabe4a8"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "4b832afc05358d696488dfd76f3e7ea5"
server_key_iv: "334d7fe81bbfe9dfce0d39e6fddd0470"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.404027 2943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/wal/instance:
uuid: "7fb81dc334b641e084097ce2feabe4a8"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "4b832afc05358d696488dfd76f3e7ea5"
server_key_iv: "334d7fe81bbfe9dfce0d39e6fddd0470"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.407265 2943 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251025 08:02:44.409406 2958 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:44.410430 2943 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:44.410540 2943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/wal
uuid: "7fb81dc334b641e084097ce2feabe4a8"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "4b832afc05358d696488dfd76f3e7ea5"
server_key_iv: "334d7fe81bbfe9dfce0d39e6fddd0470"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.410619 2943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:44.421801 2943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:44.424538 2943 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:44.424687 2943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:44.431152 2943 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.61:34389
I20251025 08:02:44.431169 3010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.61:34389 every 8 connection(s)
I20251025 08:02:44.432212 2943 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data/info.pb
I20251025 08:02:44.434978 3011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:44.438782 3011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } has no permanent_uuid. Determining permanent_uuid...
I20251025 08:02:44.439589 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 2943
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379364, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.61@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.132.61@KRBTEST.COM
I20251025 08:02:44.450933 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.440624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.27.132.62:42629)
1025 08:02:44.440761 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.440767 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.441498 (+ 731us) server_negotiation.cc:408] Connection header received
1025 08:02:44.442097 (+ 599us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.442099 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.442149 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.442218 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.443273 (+ 1055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.443822 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.444531 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.444730 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.447122 (+ 2392us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.447140 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.447142 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.447170 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.448842 (+ 1672us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.449328 (+ 486us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.449334 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.449336 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.449396 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.449730 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.449733 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.449735 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.449914 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.450015 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.450657 (+ 642us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.450785 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 08:02:44.451632 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.440853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:34968)
1025 08:02:44.441342 (+ 489us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.441381 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.441905 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.442280 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.442285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.442788 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.443104 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.443123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.443967 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.443971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.444411 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.444418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.444577 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.445305 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.445327 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.446981 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.448969 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.448976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.448983 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.449230 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.449514 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.449517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.449518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.449616 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.450022 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.450028 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.450307 (+ 279us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.450529 (+ 222us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.450812 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"spinlock_wait_cycles":34304,"thread_start_us":144,"threads_started":1}
I20251025 08:02:44.451648 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.440852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.27.132.61:34389)
1025 08:02:44.441245 (+ 393us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.441255 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.441292 (+ 37us) server_negotiation.cc:408] Connection header received
1025 08:02:44.441894 (+ 602us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.441919 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.442142 (+ 223us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.442310 (+ 168us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.443166 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.443978 (+ 812us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.444651 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.444889 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.447619 (+ 2730us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.447640 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.447649 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.447675 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.449617 (+ 1942us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.450006 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.450011 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.450013 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.450067 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.450284 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.450286 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.450287 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.450520 (+ 233us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.450693 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.451304 (+ 611us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.451487 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20251025 08:02:44.452896 3011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } has no permanent_uuid. Determining permanent_uuid...
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379364, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.61@KRBTEST.COM for oryx/127.27.132.61@KRBTEST.COM
I20251025 08:02:44.460386 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.453361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.61:34389 (local address 127.0.0.1:52400)
1025 08:02:44.453513 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.453527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.453611 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.453946 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.453949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.454084 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.454240 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.454245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.455159 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.455162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.455542 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.455550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.455646 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.456134 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.456146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.457322 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.459116 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.459120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.459122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.459317 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.459553 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.459557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.459559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.459611 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.459929 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.459933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.460014 (+ 81us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.460113 (+ 99us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.460219 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20251025 08:02:44.460417 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.453412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.27.132.61:34389)
1025 08:02:44.453514 (+ 102us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.453519 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.453571 (+ 52us) server_negotiation.cc:408] Connection header received
1025 08:02:44.453808 (+ 237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.453812 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.453853 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.453921 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.454334 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.455052 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.455648 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.455840 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.457465 (+ 1625us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.457479 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.457481 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.457502 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.459011 (+ 1509us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.459423 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.459426 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.459427 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.459465 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.459707 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.459709 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.459710 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.459840 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.459917 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.460207 (+ 290us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.460295 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251025 08:02:44.463861 3011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: Bootstrap starting.
I20251025 08:02:44.466180 3011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:44.466861 3011 log.cc:826] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:44.468540 3011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: No bootstrap required, opened a new log
I20251025 08:02:44.470940 3011 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } }
I20251025 08:02:44.471103 3011 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:44.471138 3011 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb81dc334b641e084097ce2feabe4a8, State: Initialized, Role: FOLLOWER
I20251025 08:02:44.471556 3011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } }
I20251025 08:02:44.471972 3017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } } }
I20251025 08:02:44.472079 3017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 08:02:44.472590 3011 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: configured and running, proceeding with master startup.
W20251025 08:02:44.477084 3028 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 08:02:44.477166 3028 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 08:02:44.478977 2943 master_runner.cc:428] Detected that this master 7fb81dc334b641e084097ce2feabe4a8 is joining an existing cluster
I20251025 08:02:44.479040 2943 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.132.61:34389
I20251025 08:02:44.488704 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.481220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.61:34389 (local address 127.0.0.1:52410)
1025 08:02:44.481745 (+ 525us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.481759 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.481863 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.482183 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.482186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.482528 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.482799 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.482806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.483478 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.483482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.484043 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.484053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.484171 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.485105 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.485121 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.485501 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.487251 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.487255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.487258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.487514 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.487729 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.487735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.487738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.487794 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.488136 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.488139 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.488224 (+ 85us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.488367 (+ 143us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.488505 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":242,"spinlock_wait_cycles":3968,"thread_start_us":131,"threads_started":1}
I20251025 08:02:44.488747 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.481316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.27.132.61:34389)
1025 08:02:44.481431 (+ 115us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.481435 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.481833 (+ 398us) server_negotiation.cc:408] Connection header received
1025 08:02:44.481989 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.481992 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.482033 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.482100 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.482913 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.483374 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.484153 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.484334 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.485625 (+ 1291us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.485637 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.485639 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.485656 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.487145 (+ 1489us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.487604 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.487607 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.487608 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.487645 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.487874 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.487876 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.487877 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.488040 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.488125 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.488397 (+ 272us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.488593 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":32}
WARNING: no policy specified for oryx/127.27.132.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.60@KRBTEST.COM" created.
I20251025 08:02:44.516631 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.481214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:34982)
1025 08:02:44.481664 (+ 450us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.481681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.481790 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.482093 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.482096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.482525 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.483224 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.483233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.484025 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.484028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.484569 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.484581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.484748 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.512014 (+ 27266us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.512043 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.512523 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.514853 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.514857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.514860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.515087 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.515466 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.515472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.515477 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.515549 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.516043 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.516046 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.516154 (+ 108us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.516268 (+ 114us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.516421 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"spinlock_wait_cycles":3456,"thread_start_us":100,"threads_started":1}
I20251025 08:02:44.516731 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.481283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.27.132.62:42629)
1025 08:02:44.481480 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.481485 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.481759 (+ 274us) server_negotiation.cc:408] Connection header received
1025 08:02:44.481932 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.481935 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.481989 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.482073 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.483383 (+ 1310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.483894 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.484751 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.484958 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.512764 (+ 27806us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.512796 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.512799 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.512837 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.514685 (+ 1848us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.515228 (+ 543us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.515235 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.515240 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.515305 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.515664 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.515668 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.515671 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.515891 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.515995 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.516363 (+ 368us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.516519 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379363, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.62@KRBTEST.COM for oryx/127.27.132.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.132.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
I20251025 08:02:44.532307 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.521726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.27.132.61:34389)
1025 08:02:44.521849 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.521853 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.522765 (+ 912us) server_negotiation.cc:408] Connection header received
1025 08:02:44.522978 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.522981 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.523041 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.523132 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.524030 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.524756 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.525390 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.525590 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.527917 (+ 2327us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.527940 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.527944 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.527974 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.530449 (+ 2475us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.531041 (+ 592us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.531044 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.531046 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.531105 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.531408 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.531412 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.531414 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.531578 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.531673 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.532031 (+ 358us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.532139 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251025 08:02:44.532310 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.521684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.61:34389 (local address 127.0.0.1:52420)
1025 08:02:44.522627 (+ 943us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.522676 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.522844 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.523164 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.523169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.523654 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.523916 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.523923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.524911 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.524915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.525294 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.525302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.525396 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.526054 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.526072 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.527783 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.530579 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.530591 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.530600 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.530935 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.531218 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.531221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.531223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.531289 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.531698 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.531705 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.531823 (+ 118us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.531950 (+ 127us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.532141 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":113,"threads_started":1}
I20251025 08:02:44.534070 2470 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.27.132.61:34389: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8"
member_type: NON_VOTER
last_known_addr {
host: "127.27.132.61"
port: 34389
}
attrs {
promote: true
}
}
dest_uuid: "d28b5159b5704014b011a9766e6f0e8c"
cas_config_opid_index: -1
I20251025 08:02:44.534771 2470 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } }
I20251025 08:02:44.536813 3041 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7fb81dc334b641e084097ce2feabe4a8 (127.27.132.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } } }
I20251025 08:02:44.537477 2985 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for HTTP/127.27.132.60@KRBTEST.COM; defaulting to no policy
I20251025 08:02:44.538450 3041 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.27.132.61:34389
I20251025 08:02:44.539028 3042 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } } }
I20251025 08:02:44.539126 3042 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:44.539170 2943 master.cc:561] Master@127.27.132.61:34389 shutting down...
I20251025 08:02:44.539531 2985 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Refusing update from remote peer d28b5159b5704014b011a9766e6f0e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251025 08:02:44.540189 3042 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.60@KRBTEST.COM" created.
I20251025 08:02:44.544595 2943 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:44.544879 2943 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:44.544999 2943 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: stopping tablet replica
W20251025 08:02:44.545785 2454 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20251025 08:02:44.547308 2454 consensus_peers.cc:597] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c -> Peer 7fb81dc334b641e084097ce2feabe4a8 (127.27.132.61:34389): Couldn't send request to peer 7fb81dc334b641e084097ce2feabe4a8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.132.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab.
I20251025 08:02:44.556380 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.60:37999
--webserver_interface=127.27.132.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:42629,127.27.132.61:34389,127.27.132.60:37999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251025 08:02:44.559867 2943 master.cc:583] Master@127.27.132.61:34389 shutdown complete.
I20251025 08:02:44.559942 2943 master_runner.cc:305] Clearing existing system tablet
I20251025 08:02:44.560213 2943 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:44.560281 2943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:44.561901 2943 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20251025 08:02:44.562976 3048 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:44.563251 2943 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:44.563444 2943 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251025 08:02:44.563489 2943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/wal
uuid: "7fb81dc334b641e084097ce2feabe4a8"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "4b832afc05358d696488dfd76f3e7ea5"
server_key_iv: "334d7fe81bbfe9dfce0d39e6fddd0470"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.565922 2943 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:44.568459 2943 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251025 08:02:44.568560 2943 log.cc:1199] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20251025 08:02:44.568842 2943 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: Deleting consensus metadata
I20251025 08:02:44.569031 2943 master_runner.cc:315] Copying system tablet from 127.27.132.62:42629
I20251025 08:02:44.570297 2943 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.132.62:42629
I20251025 08:02:44.578711 3054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.570904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:34984)
1025 08:02:44.571225 (+ 321us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.571239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.571391 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.571711 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.571715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.571934 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.572237 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.572245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.573254 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.573258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.573920 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.573931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.574050 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.574670 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.574687 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.575157 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.577245 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.577249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.577251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.577473 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.577745 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.577749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.577751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.577803 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.578149 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.578152 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.578249 (+ 97us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.578358 (+ 109us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.578494 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":92,"threads_started":1}
I20251025 08:02:44.578725 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.571044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.27.132.62:42629)
1025 08:02:44.571186 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.571191 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.571358 (+ 167us) server_negotiation.cc:408] Connection header received
1025 08:02:44.571517 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.571525 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.571588 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.571676 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.572377 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.573129 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.574042 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.574266 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.575302 (+ 1036us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.575351 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.575355 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.575384 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.577120 (+ 1736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.577590 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.577594 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.577596 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.577648 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.577885 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.577888 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.577890 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.578062 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.578158 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.578420 (+ 262us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.578537 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251025 08:02:44.579787 2490 tablet_copy_service.cc:140] P d28b5159b5704014b011a9766e6f0e8c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7fb81dc334b641e084097ce2feabe4a8 ({username='oryx', principal='oryx/127.27.132.61@KRBTEST.COM'} at 127.0.0.1:34984)
I20251025 08:02:44.579943 2490 tablet_copy_service.cc:161] P d28b5159b5704014b011a9766e6f0e8c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7fb81dc334b641e084097ce2feabe4a8 at {username='oryx', principal='oryx/127.27.132.61@KRBTEST.COM'} at 127.0.0.1:34984: session id = 7fb81dc334b641e084097ce2feabe4a8-00000000000000000000000000000000
I20251025 08:02:44.581514 2490 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c: Tablet Copy: opened 0 blocks and 1 log segments
I20251025 08:02:44.583220 2943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251025 08:02:44.585541 2943 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: tablet copy: Starting download of 0 data blocks...
I20251025 08:02:44.585733 2943 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: tablet copy: Starting download of 1 WAL segments...
I20251025 08:02:44.587688 2943 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251025 08:02:44.589576 2490 tablet_copy_service.cc:342] P d28b5159b5704014b011a9766e6f0e8c: Request end of tablet copy session 7fb81dc334b641e084097ce2feabe4a8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.132.61@KRBTEST.COM'} at 127.0.0.1:34984
I20251025 08:02:44.589706 2490 tablet_copy_service.cc:434] P d28b5159b5704014b011a9766e6f0e8c: ending tablet copy session 7fb81dc334b641e084097ce2feabe4a8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7fb81dc334b641e084097ce2feabe4a8
W20251025 08:02:44.594913 2943 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251025 08:02:44.597299 2943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:44.599036 3055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:44.599056 3056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:44.599484 2943 server_base.cc:1047] running on GCE node
W20251025 08:02:44.599283 3058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:44.599722 2943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:44.600013 2943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:44.601150 2943 hybrid_clock.cc:648] HybridClock initialized: now 1761379364601133 us; error 34 us; skew 500 ppm
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379364, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:44.602829 2943 init.cc:377] Logged in from keytab as oryx/127.27.132.61@KRBTEST.COM (short username oryx)
I20251025 08:02:44.603307 2943 webserver.cc:492] Webserver started at http://127.27.132.61:42431/ using document root <none> and password file <none>
I20251025 08:02:44.603526 2943 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:44.603574 2943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:44.604871 2943 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251025 08:02:44.605870 3065 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:44.606132 2943 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251025 08:02:44.606243 2943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/wal
uuid: "7fb81dc334b641e084097ce2feabe4a8"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "4b832afc05358d696488dfd76f3e7ea5"
server_key_iv: "334d7fe81bbfe9dfce0d39e6fddd0470"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.606390 2943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:44.620381 2943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:44.622793 2943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:44.627943 2943 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.61:34389
I20251025 08:02:44.628100 3117 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.61:34389 every 8 connection(s)
I20251025 08:02:44.628587 2943 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-1/data/info.pb
I20251025 08:02:44.630450 3118 sys_catalog.cc:263] Verifying existing consensus state
I20251025 08:02:44.631534 3118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: Bootstrap starting.
I20251025 08:02:44.641848 3118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251025 08:02:44.642218 3118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: Bootstrap complete.
I20251025 08:02:44.642944 3118 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } }
I20251025 08:02:44.643075 3118 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7fb81dc334b641e084097ce2feabe4a8, State: Initialized, Role: LEARNER
I20251025 08:02:44.643177 3118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } }
I20251025 08:02:44.643663 3121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } } }
I20251025 08:02:44.643850 3118 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:44.643851 3121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: This master's current role is: LEARNER
I20251025 08:02:44.647717 3132 catalog_manager.cc:1269] Loaded cluster ID: 73f659919e404a6591f3f9e2aa29e7c7
I20251025 08:02:44.647792 3132 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: loading cluster ID for follower catalog manager: success
I20251025 08:02:44.650077 3132 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: acquiring CA information for follower catalog manager: success
I20251025 08:02:44.651888 3132 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251025 08:02:44.685624 3046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:44.685889 3046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:44.685977 3046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:44.689369 3046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:44.689441 3046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:44.689483 3046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:44.689523 3046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:44.689563 3046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:44.693917 3046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:42629,127.27.132.61:34389,127.27.132.60:37999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.60:37999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.132.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:44.695024 3046 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:44.695919 3046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:44.701499 3137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:44.701527 3138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:44.701499 3140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:44.701586 3046 server_base.cc:1047] running on GCE node
I20251025 08:02:44.702175 3046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:44.703120 3046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:44.704268 3046 hybrid_clock.cc:648] HybridClock initialized: now 1761379364704256 us; error 33 us; skew 500 ppm
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379364, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:44.707190 3046 init.cc:377] Logged in from keytab as oryx/127.27.132.60@KRBTEST.COM (short username oryx)
I20251025 08:02:44.708292 3046 webserver.cc:492] Webserver started at http://127.27.132.60:46213/ using document root <none> and password file <none>
I20251025 08:02:44.708817 3046 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:44.708864 3046 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:44.709030 3046 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:44.710664 3046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data/instance:
uuid: "26d7cf36796744208fc9d9bf1d960347"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "c1891768dfe7a9e33eeec11354e4ab4c"
server_key_iv: "d617ae2bed57db3ca5143db7c6e934e5"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.711087 3046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/wal/instance:
uuid: "26d7cf36796744208fc9d9bf1d960347"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "c1891768dfe7a9e33eeec11354e4ab4c"
server_key_iv: "d617ae2bed57db3ca5143db7c6e934e5"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.714404 3046 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20251025 08:02:44.716513 3147 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:44.717581 3046 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:44.717685 3046 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/wal
uuid: "26d7cf36796744208fc9d9bf1d960347"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "c1891768dfe7a9e33eeec11354e4ab4c"
server_key_iv: "d617ae2bed57db3ca5143db7c6e934e5"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.717792 3046 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:44.735996 3046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:44.739398 3046 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:44.739578 3046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:44.746687 3199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.60:37999 every 8 connection(s)
I20251025 08:02:44.746697 3046 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.60:37999
I20251025 08:02:44.747807 3046 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data/info.pb
I20251025 08:02:44.750563 3200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:44.753000 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 3046
I20251025 08:02:44.754871 3200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } has no permanent_uuid. Determining permanent_uuid...
I20251025 08:02:44.759212 2663 heartbeater.cc:499] Master 127.27.132.62:42629 was elected leader, sending a full tablet report...
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.132.60@KRBTEST.COM
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379364, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.60@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:44.766245 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.756873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.27.132.62:42629)
1025 08:02:44.757012 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.757017 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.757187 (+ 170us) server_negotiation.cc:408] Connection header received
1025 08:02:44.757336 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.757339 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.757383 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.757461 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.758298 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.758825 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.759778 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.759973 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.762785 (+ 2812us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.762804 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.762806 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.762836 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.764460 (+ 1624us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.764978 (+ 518us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.764981 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.764983 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.765031 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.765294 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.765298 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.765300 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.765466 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.765561 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.765943 (+ 382us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.766069 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251025 08:02:44.766937 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.754480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.27.132.60:37999)
1025 08:02:44.754943 (+ 463us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.754956 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.755000 (+ 44us) server_negotiation.cc:408] Connection header received
1025 08:02:44.755681 (+ 681us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.755706 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.756066 (+ 360us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.756391 (+ 325us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.757483 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.758270 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.759001 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.759704 (+ 703us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.762263 (+ 2559us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.762288 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.762302 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.762327 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.764480 (+ 2153us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.764943 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.764948 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.764949 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.765015 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.765250 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.765253 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.765255 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.765601 (+ 346us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.765780 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.765990 (+ 210us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.766177 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":148,"threads_started":1}
I20251025 08:02:44.767444 3203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.756689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:34992)
1025 08:02:44.757075 (+ 386us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.757110 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.757229 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.757531 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.757535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.757902 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.758154 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.758164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.758959 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.758966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.759615 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.759627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.760120 (+ 493us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.760675 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.760712 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.762633 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.764632 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.764641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.764652 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.764863 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.765123 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.765126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.765128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.765199 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.765574 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.765583 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.765712 (+ 129us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.765895 (+ 183us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.767280 (+ 1385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":1127,"thread_start_us":128,"threads_started":1}
I20251025 08:02:44.768956 3200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } has no permanent_uuid. Determining permanent_uuid...
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379364, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.60@KRBTEST.COM for oryx/127.27.132.61@KRBTEST.COM
I20251025 08:02:44.777161 3203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.769422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.61:34389 (local address 127.0.0.1:52424)
1025 08:02:44.769563 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.769575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.769642 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.770060 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.770063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.770225 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.770427 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.770446 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.771380 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.771383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.771785 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.771792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.771898 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.772427 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.772453 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.773715 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.775656 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.775659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.775661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.775828 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.776073 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.776076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.776077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.776123 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.776608 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.776613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.776714 (+ 101us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.776820 (+ 106us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.776962 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20251025 08:02:44.777247 3204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.769596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.27.132.61:34389)
1025 08:02:44.769855 (+ 259us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.769859 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.769870 (+ 11us) server_negotiation.cc:408] Connection header received
1025 08:02:44.769904 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.769906 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.769948 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.770035 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.770562 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.771246 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.771896 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.772110 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.773844 (+ 1734us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.773861 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.773864 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.773888 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.775544 (+ 1656us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.775933 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.775936 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.775937 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.775982 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.776252 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.776258 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.776262 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.776497 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.776628 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.776904 (+ 276us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.777063 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
I20251025 08:02:44.777981 3200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } has no permanent_uuid. Determining permanent_uuid...
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379364, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.60@KRBTEST.COM for oryx/127.27.132.60@KRBTEST.COM
I20251025 08:02:44.785460 3203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.778463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.60:37999 (local address 127.0.0.1:37836)
1025 08:02:44.778577 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.778593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.778673 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.778921 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.778924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.779055 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.779216 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.779221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.779996 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.779998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.780323 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.780330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.780417 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.780960 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.780972 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.782211 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.784237 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.784240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.784242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.784413 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.784609 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.784612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.784613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.784652 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.785017 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.785020 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.785095 (+ 75us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.785181 (+ 86us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.785278 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20251025 08:02:44.785467 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.778630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.27.132.60:37999)
1025 08:02:44.778755 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.778759 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.778771 (+ 12us) server_negotiation.cc:408] Connection header received
1025 08:02:44.778798 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.778800 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.778834 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.778901 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.779424 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.779890 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.780455 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.780631 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.782348 (+ 1717us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.782364 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.782366 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.782409 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.784112 (+ 1703us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.784507 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.784509 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.784511 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.784546 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.784771 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.784774 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.784775 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.784928 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.785019 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.785231 (+ 212us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.785329 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251025 08:02:44.788405 3200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: Bootstrap starting.
I20251025 08:02:44.790901 3200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:44.791697 3200 log.cc:826] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:44.793612 3200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: No bootstrap required, opened a new log
I20251025 08:02:44.796324 3200 raft_consensus.cc:359] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } }
I20251025 08:02:44.796511 3200 raft_consensus.cc:385] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:44.796551 3200 raft_consensus.cc:740] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d7cf36796744208fc9d9bf1d960347, State: Initialized, Role: FOLLOWER
I20251025 08:02:44.796983 3200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } }
I20251025 08:02:44.797470 3206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } } }
I20251025 08:02:44.797585 3206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 08:02:44.798102 3200 sys_catalog.cc:565] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: configured and running, proceeding with master startup.
W20251025 08:02:44.802836 3217 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 08:02:44.802935 3217 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 08:02:44.806758 3046 master_runner.cc:428] Detected that this master 26d7cf36796744208fc9d9bf1d960347 is joining an existing cluster
I20251025 08:02:44.806850 3046 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.132.60:37999
I20251025 08:02:44.824817 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.821665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:35008)
1025 08:02:44.821876 (+ 211us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.821892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.821990 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.822270 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.822273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.822290 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:44.822540 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.822548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.823318 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.823348 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.824319 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.824329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.824456 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.824533 (+ 77us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.824588 (+ 55us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.824660 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20251025 08:02:44.825268 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.821681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.27.132.62:42629)
1025 08:02:44.822063 (+ 382us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.822066 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.822079 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:02:44.822127 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.822129 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.822171 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.822248 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:44.822689 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.823218 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.824460 (+ 1242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.824807 (+ 347us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.824866 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.824916 (+ 50us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.825136 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"spinlock_wait_cycles":1408,"thread_start_us":157,"threads_started":1}
I20251025 08:02:44.829771 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.826457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.61:34389 (local address 127.0.0.1:52450)
1025 08:02:44.826603 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.826617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.826715 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.827184 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.827187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.827206 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:44.827416 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.827427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.828533 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.828536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.829416 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.829428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.829507 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.829527 (+ 20us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.829563 (+ 36us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.829625 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20251025 08:02:44.831068 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.826588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.27.132.61:34389)
1025 08:02:44.826910 (+ 322us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.826915 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.826930 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:02:44.826981 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.826985 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.827048 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.827160 (+ 112us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:44.827571 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.828391 (+ 820us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.830095 (+ 1704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.830606 (+ 511us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.830781 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.830839 (+ 58us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.830914 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":128,"threads_started":1}
I20251025 08:02:44.836664 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.810579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.60:37999 (local address 127.0.0.1:37846)
1025 08:02:44.810854 (+ 275us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.810868 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.810975 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.811221 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.811223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.827667 (+ 16444us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.828005 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.828016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.828945 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.828948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.829479 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.829487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.829591 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.830878 (+ 1287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.830895 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.831637 (+ 742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.834281 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.834284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.834286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.834494 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.835408 (+ 914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.835411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.835413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.835466 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.836174 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.836177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.836248 (+ 71us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.836337 (+ 89us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.836462 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"spinlock_wait_cycles":3840,"thread_start_us":82,"threads_started":1}
I20251025 08:02:44.837391 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.809492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.61:34389 (local address 127.0.0.1:52440)
1025 08:02:44.809833 (+ 341us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.809850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.809955 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.810445 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.810449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.827677 (+ 17228us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.828369 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.828380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.829384 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.829398 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.829954 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.829967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.830071 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.830891 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.830906 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.831736 (+ 830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.834125 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.834129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.834131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.834590 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.834975 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.834978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.834981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.835023 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.836912 (+ 1889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.836916 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.837004 (+ 88us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.837107 (+ 103us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.837227 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"spinlock_wait_cycles":15488,"thread_start_us":116,"threads_started":1}
I20251025 08:02:44.837880 3204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.809936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.27.132.61:34389)
1025 08:02:44.810082 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.810087 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.810102 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:02:44.810149 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.810153 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.810207 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.810296 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.828529 (+ 18233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.829250 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.830592 (+ 1342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.830729 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.831872 (+ 1143us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.831888 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.831891 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.831918 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.834008 (+ 2090us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.834692 (+ 684us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.834697 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.834699 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.834751 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.835108 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.835111 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.835113 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.835284 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.836876 (+ 1592us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.837585 (+ 709us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.837725 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251025 08:02:44.839998 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.810469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.27.132.60:37999)
1025 08:02:44.810567 (+ 98us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.810573 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.810925 (+ 352us) server_negotiation.cc:408] Connection header received
1025 08:02:44.811077 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.811081 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.811124 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.811203 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.828148 (+ 16945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.828819 (+ 671us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.830061 (+ 1242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.830558 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.831767 (+ 1209us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.831783 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.831787 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.831809 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.834165 (+ 2356us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.834849 (+ 684us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.834853 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.834855 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.834893 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.835552 (+ 659us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.835555 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.835557 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.836046 (+ 489us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.836140 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.838064 (+ 1924us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.838170 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20251025 08:02:44.842084 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.809722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.27.132.62:42629)
1025 08:02:44.809894 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.809901 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.810415 (+ 514us) server_negotiation.cc:408] Connection header received
1025 08:02:44.810725 (+ 310us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.810729 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.810785 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.810874 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.833125 (+ 22251us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.833925 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.835080 (+ 1155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.835269 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.836475 (+ 1206us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.836490 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.836494 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.836519 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.838892 (+ 2373us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.840612 (+ 1720us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.840616 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.840618 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.840669 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.841223 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.841227 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.841229 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.841420 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.841517 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.841811 (+ 294us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.841933 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20251025 08:02:44.842312 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.809492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:35000)
1025 08:02:44.810318 (+ 826us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.810334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.810601 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.810919 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.810923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.832128 (+ 21205us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.832979 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.832993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.834375 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.834379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.834953 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.834962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.835431 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.835957 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.835972 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.836348 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.839115 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.839119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.839120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.840496 (+ 1376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.840946 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.840949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.840951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.841002 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.841517 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.841521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.841609 (+ 88us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.841709 (+ 100us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.841849 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":21,"spinlock_wait_cycles":1920,"thread_start_us":134,"threads_started":1}
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379363, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.62@KRBTEST.COM for oryx/127.27.132.60@KRBTEST.COM
I20251025 08:02:44.853415 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.846000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.27.132.60:37999)
1025 08:02:44.846120 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.846125 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.846138 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:02:44.846274 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.846277 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.846332 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.846416 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.846992 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.847596 (+ 604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.848581 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.848788 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.850411 (+ 1623us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.850428 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.850430 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.850458 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.852033 (+ 1575us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.852482 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.852484 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.852486 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.852521 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.852742 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.852744 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.852745 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.852882 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.852953 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.853201 (+ 248us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.853294 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 08:02:44.853480 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.845906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.60:37999 (local address 127.0.0.1:37848)
1025 08:02:44.846087 (+ 181us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.846100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.846167 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.846458 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.846461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.846678 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.846874 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.846881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.847708 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.847711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.848254 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.848262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.848370 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.848885 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.848903 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.850271 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.852151 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.852156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.852158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.852391 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.852594 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.852597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.852600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.852654 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.852962 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.852965 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.853062 (+ 97us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.853171 (+ 109us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.853311 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20251025 08:02:44.854432 2470 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.27.132.60:37999: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "26d7cf36796744208fc9d9bf1d960347"
member_type: NON_VOTER
last_known_addr {
host: "127.27.132.60"
port: 37999
}
attrs {
promote: true
}
}
dest_uuid: "d28b5159b5704014b011a9766e6f0e8c"
cas_config_opid_index: 5
I20251025 08:02:44.855008 2470 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } }
I20251025 08:02:44.856103 3042 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 26d7cf36796744208fc9d9bf1d960347 (127.27.132.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:44.856496 3092 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 LEARNER]: Refusing update from remote peer d28b5159b5704014b011a9766e6f0e8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251025 08:02:44.856705 3174 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:44.857759 3042 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.27.132.60:37999
I20251025 08:02:44.857959 3042 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251025 08:02:44.858122 3174 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 FOLLOWER]: Refusing update from remote peer d28b5159b5704014b011a9766e6f0e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251025 08:02:44.858245 3042 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:44.858321 3042 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:44.858592 3046 master.cc:561] Master@127.27.132.60:37999 shutting down...
I20251025 08:02:44.860119 3228 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:44.859933 3092 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 26d7cf36796744208fc9d9bf1d960347 (127.27.132.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:44.862211 3121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28b5159b5704014b011a9766e6f0e8c. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:44.862347 3121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: This master's current role is: LEARNER
I20251025 08:02:44.862432 3230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: NON_VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: true } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:44.862534 3230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: This master's current role is: LEARNER
W20251025 08:02:44.863052 2454 consensus_peers.cc:597] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c -> Peer 26d7cf36796744208fc9d9bf1d960347 (127.27.132.60:37999): Couldn't send request to peer 26d7cf36796744208fc9d9bf1d960347. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251025 08:02:44.863152 3228 raft_consensus.cc:1064] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c: attempting to promote NON_VOTER 7fb81dc334b641e084097ce2feabe4a8 to VOTER
I20251025 08:02:44.863710 3228 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } }
I20251025 08:02:44.865106 3092 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 LEARNER]: Refusing update from remote peer d28b5159b5704014b011a9766e6f0e8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20251025 08:02:44.865679 3041 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251025 08:02:44.865933 3046 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 08:02:44.866057 2454 consensus_peers.cc:597] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c -> Peer 26d7cf36796744208fc9d9bf1d960347 (127.27.132.60:37999): Couldn't send request to peer 26d7cf36796744208fc9d9bf1d960347. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
W20251025 08:02:44.866058 2470 master.cc:679] Remote error: unable to get registration information for peer 26d7cf36796744208fc9d9bf1d960347: Service unavailable: service kudu.master.MasterService not registered on Master
I20251025 08:02:44.866221 3046 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:44.866331 3046 tablet_replica.cc:333] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: stopping tablet replica
I20251025 08:02:44.869922 3228 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7fb81dc334b641e084097ce2feabe4a8 (127.27.132.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:44.870522 3092 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7fb81dc334b641e084097ce2feabe4a8 (127.27.132.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:44.870864 3046 master.cc:583] Master@127.27.132.60:37999 shutdown complete.
I20251025 08:02:44.870920 3046 master_runner.cc:305] Clearing existing system tablet
I20251025 08:02:44.871150 3046 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:44.871215 3046 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:44.871769 3228 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:44.871882 3228 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:44.872041 3042 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:44.872136 3042 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:44.872382 3230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:44.872510 3230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 08:02:44.872857 3046 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251025 08:02:44.873668 3232 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:44.873908 3046 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:44.874006 3046 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251025 08:02:44.874053 3046 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/wal
uuid: "26d7cf36796744208fc9d9bf1d960347"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "c1891768dfe7a9e33eeec11354e4ab4c"
server_key_iv: "d617ae2bed57db3ca5143db7c6e934e5"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.875870 3046 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:44.877502 3046 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251025 08:02:44.877574 3046 log.cc:1199] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20251025 08:02:44.877799 3046 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: Deleting consensus metadata
I20251025 08:02:44.877979 3046 master_runner.cc:315] Copying system tablet from 127.27.132.62:42629
I20251025 08:02:44.878988 3046 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: tablet copy: Beginning tablet copy session from remote peer at address 127.27.132.62:42629
I20251025 08:02:44.887508 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.879631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.27.132.62:42629)
1025 08:02:44.879826 (+ 195us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.879830 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.879917 (+ 87us) server_negotiation.cc:408] Connection header received
1025 08:02:44.880061 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.880063 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.880108 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.880179 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:44.880880 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.881380 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.882195 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.882321 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.883953 (+ 1632us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:44.883972 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:44.883974 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:44.883999 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:44.885773 (+ 1774us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.886268 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.886271 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.886273 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.886327 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:44.886599 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:44.886602 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:44.886604 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:44.886792 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:44.886891 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.887202 (+ 311us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.887351 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251025 08:02:44.887542 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.879566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:35014)
1025 08:02:44.879833 (+ 267us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.879850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.879969 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.880211 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.880215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.880446 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:44.880743 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.880752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.881481 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.881485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.882095 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.882104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.882220 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.883363 (+ 1143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:44.883381 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:44.883803 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:44.885909 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.885912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.885914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.886131 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.886429 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:44.886432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:44.886435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:44.886495 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:44.886938 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:44.886940 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:44.887025 (+ 85us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.887139 (+ 114us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.887299 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20251025 08:02:44.887811 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.887437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.60:37999 (local address 127.0.0.1:37850)
1025 08:02:44.887612 (+ 175us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.887739 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.60:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20251025 08:02:44.888171 2470 master.cc:679] Network error: unable to get registration information for peer 26d7cf36796744208fc9d9bf1d960347: Client connection negotiation failed: client connection to 127.27.132.60:37999: connect: Connection refused (error 111)
I20251025 08:02:44.888334 2490 tablet_copy_service.cc:140] P d28b5159b5704014b011a9766e6f0e8c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 26d7cf36796744208fc9d9bf1d960347 ({username='oryx', principal='oryx/127.27.132.60@KRBTEST.COM'} at 127.0.0.1:35014)
I20251025 08:02:44.888422 2490 tablet_copy_service.cc:161] P d28b5159b5704014b011a9766e6f0e8c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 26d7cf36796744208fc9d9bf1d960347 at {username='oryx', principal='oryx/127.27.132.60@KRBTEST.COM'} at 127.0.0.1:35014: session id = 26d7cf36796744208fc9d9bf1d960347-00000000000000000000000000000000
I20251025 08:02:44.889148 2490 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c: Tablet Copy: opened 0 blocks and 1 log segments
I20251025 08:02:44.890255 3046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251025 08:02:44.892086 3046 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: tablet copy: Starting download of 0 data blocks...
I20251025 08:02:44.892251 3046 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: tablet copy: Starting download of 1 WAL segments...
I20251025 08:02:44.894011 3046 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251025 08:02:44.895629 2490 tablet_copy_service.cc:342] P d28b5159b5704014b011a9766e6f0e8c: Request end of tablet copy session 26d7cf36796744208fc9d9bf1d960347-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.132.60@KRBTEST.COM'} at 127.0.0.1:35014
I20251025 08:02:44.895713 2490 tablet_copy_service.cc:434] P d28b5159b5704014b011a9766e6f0e8c: ending tablet copy session 26d7cf36796744208fc9d9bf1d960347-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 26d7cf36796744208fc9d9bf1d960347
W20251025 08:02:44.899926 3046 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251025 08:02:44.902114 3046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:44.903810 3239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:44.903792 3242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:44.903885 3240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:44.904119 3046 server_base.cc:1047] running on GCE node
I20251025 08:02:44.904263 3046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:44.904572 3046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:44.905746 3046 hybrid_clock.cc:648] HybridClock initialized: now 1761379364905726 us; error 38 us; skew 500 ppm
Oct 25 08:02:44 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379364, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:44.907585 3046 init.cc:377] Logged in from keytab as oryx/127.27.132.60@KRBTEST.COM (short username oryx)
I20251025 08:02:44.908141 3046 webserver.cc:492] Webserver started at http://127.27.132.60:41667/ using document root <none> and password file <none>
I20251025 08:02:44.908339 3046 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:44.908375 3046 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:44.909478 3046 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251025 08:02:44.910360 3249 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:44.910640 3046 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251025 08:02:44.910701 3046 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/wal
uuid: "26d7cf36796744208fc9d9bf1d960347"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "c1891768dfe7a9e33eeec11354e4ab4c"
server_key_iv: "d617ae2bed57db3ca5143db7c6e934e5"
server_key_version: "encryptionkey@0"
I20251025 08:02:44.910785 3046 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:44.919019 3046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:44.921053 3046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:44.924466 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.924171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.60:37999 (local address 127.0.0.1:37856)
1025 08:02:44.924330 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.924379 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.60:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20251025 08:02:44.924695 2470 master.cc:679] Network error: unable to get registration information for peer 26d7cf36796744208fc9d9bf1d960347: Client connection negotiation failed: client connection to 127.27.132.60:37999: connect: Connection refused (error 111)
I20251025 08:02:44.925949 3046 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.60:37999
I20251025 08:02:44.926128 3301 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.60:37999 every 8 connection(s)
I20251025 08:02:44.926571 3046 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/master-2/data/info.pb
I20251025 08:02:44.928046 3302 sys_catalog.cc:263] Verifying existing consensus state
I20251025 08:02:44.928834 3302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: Bootstrap starting.
I20251025 08:02:44.937669 3302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251025 08:02:44.937963 3302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: Bootstrap complete.
I20251025 08:02:44.938513 3302 raft_consensus.cc:359] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } }
I20251025 08:02:44.938622 3302 raft_consensus.cc:740] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 26d7cf36796744208fc9d9bf1d960347, State: Initialized, Role: LEARNER
I20251025 08:02:44.938719 3302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } }
I20251025 08:02:44.939050 3305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:44.939150 3305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: This master's current role is: LEARNER
I20251025 08:02:44.939225 3302 sys_catalog.cc:565] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:44.942114 3316 catalog_manager.cc:1269] Loaded cluster ID: 73f659919e404a6591f3f9e2aa29e7c7
I20251025 08:02:44.942171 3316 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: loading cluster ID for follower catalog manager: success
I20251025 08:02:44.944226 3316 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: acquiring CA information for follower catalog manager: success
I20251025 08:02:44.945441 3316 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 08:02:44.995429 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.992621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.60:37999 (local address 127.0.0.1:37868)
1025 08:02:44.992791 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:44.992806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:44.992899 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:44.993256 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:44.993259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:44.993278 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:44.993471 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.993477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.994384 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.994387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:44.994995 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:44.995003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.995111 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.995135 (+ 24us) client_negotiation.cc:769] Sending connection context
1025 08:02:44.995188 (+ 53us) client_negotiation.cc:240] Negotiation successful
1025 08:02:44.995255 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20251025 08:02:44.995970 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:44.992738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.27.132.60:37999)
1025 08:02:44.993029 (+ 291us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:44.993033 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:44.993044 (+ 11us) server_negotiation.cc:408] Connection header received
1025 08:02:44.993084 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:44.993087 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:44.993137 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:44.993234 (+ 97us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:44.993584 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.994267 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:44.995111 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:44.995646 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:44.995779 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:44.995826 (+ 47us) server_negotiation.cc:300] Negotiation successful
1025 08:02:44.995875 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":109,"threads_started":1}
I20251025 08:02:45.002372 2799 heartbeater.cc:499] Master 127.27.132.62:42629 was elected leader, sending a full tablet report...
I20251025 08:02:45.223857 2936 heartbeater.cc:499] Master 127.27.132.62:42629 was elected leader, sending a full tablet report...
I20251025 08:02:45.434120 3271 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20251025 08:02:45.434540 3305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28b5159b5704014b011a9766e6f0e8c. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:45.434700 3305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: This master's current role is: LEARNER
I20251025 08:02:45.840664 3318 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: NON_VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: true } } }
I20251025 08:02:45.840842 3318 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:45.841792 3318 raft_consensus.cc:1064] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c: attempting to promote NON_VOTER 26d7cf36796744208fc9d9bf1d960347 to VOTER
I20251025 08:02:45.842144 3318 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } }
I20251025 08:02:45.843425 3271 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 LEARNER]: Refusing update from remote peer d28b5159b5704014b011a9766e6f0e8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251025 08:02:45.843433 3092 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Refusing update from remote peer d28b5159b5704014b011a9766e6f0e8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251025 08:02:45.843833 3228 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251025 08:02:45.844053 3319 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251025 08:02:45.845695 3319 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 26d7cf36796744208fc9d9bf1d960347 (127.27.132.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:45.846185 3092 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 26d7cf36796744208fc9d9bf1d960347 (127.27.132.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:45.846983 3319 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:45.847096 3319 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:45.847064 3228 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:45.847174 3228 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:45.847455 3322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:45.847572 3322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 08:02:45.848349 3228 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:45.848433 3228 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:45.848819 3271 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 26d7cf36796744208fc9d9bf1d960347 (127.27.132.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:45.850100 3305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d28b5159b5704014b011a9766e6f0e8c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:45.850214 3305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 08:02:45.916370 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.906847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.27.132.61:34389)
1025 08:02:45.907157 (+ 310us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:45.907160 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:45.907174 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:02:45.907253 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:45.907256 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:45.907298 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:45.907408 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:45.908189 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.908726 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.909322 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.909485 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.913412 (+ 3927us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:45.913430 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:45.913432 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:45.913466 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:45.915047 (+ 1581us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:45.915438 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:45.915441 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:45.915452 (+ 11us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:45.915489 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:45.915665 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:45.915667 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:45.915668 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:45.915815 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:45.915904 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:45.916076 (+ 172us) server_negotiation.cc:300] Negotiation successful
1025 08:02:45.916165 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
I20251025 08:02:45.919793 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.917122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:35020)
1025 08:02:45.917371 (+ 249us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:45.917385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:45.917462 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:45.917716 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:45.917720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:45.917739 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:45.917906 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.917911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.918648 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.918652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:45.919463 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.919473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.919582 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.919599 (+ 17us) client_negotiation.cc:769] Sending connection context
1025 08:02:45.919638 (+ 39us) client_negotiation.cc:240] Negotiation successful
1025 08:02:45.919686 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
I20251025 08:02:45.920158 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.917214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.27.132.62:42629)
1025 08:02:45.917436 (+ 222us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:45.917439 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:45.917462 (+ 23us) server_negotiation.cc:408] Connection header received
1025 08:02:45.917537 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:45.917539 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:45.917598 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:45.917669 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:45.918002 (+ 333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.918529 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.919580 (+ 1051us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.919914 (+ 334us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.919954 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:45.920007 (+ 53us) server_negotiation.cc:300] Negotiation successful
1025 08:02:45.920064 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":74,"threads_started":1}
I20251025 08:02:45.923481 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.921275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.61:34389 (local address 127.0.0.1:52470)
1025 08:02:45.921410 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:45.921425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:45.921503 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:45.921724 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:45.921728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:45.921741 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:45.921887 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.921892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.922544 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.922548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:45.923117 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.923124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.923227 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.923244 (+ 17us) client_negotiation.cc:769] Sending connection context
1025 08:02:45.923295 (+ 51us) client_negotiation.cc:240] Negotiation successful
1025 08:02:45.923368 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20251025 08:02:45.923882 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.921339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.27.132.61:34389)
1025 08:02:45.921457 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:45.921462 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:45.921475 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:02:45.921601 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:45.921604 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:45.921634 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:45.921714 (+ 80us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:45.921981 (+ 267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.922448 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.923223 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.923698 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.923727 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:45.923770 (+ 43us) server_negotiation.cc:300] Negotiation successful
1025 08:02:45.923811 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251025 08:02:45.927615 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.924942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.60:37999 (local address 127.0.0.1:37878)
1025 08:02:45.925083 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:45.925098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:45.925181 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:45.925526 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:45.925529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:45.925543 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:45.925686 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.925691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.926649 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.926653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:45.927224 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.927231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.927363 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.927385 (+ 22us) client_negotiation.cc:769] Sending connection context
1025 08:02:45.927442 (+ 57us) client_negotiation.cc:240] Negotiation successful
1025 08:02:45.927505 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20251025 08:02:45.928282 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.925085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.27.132.60:37999)
1025 08:02:45.925328 (+ 243us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:45.925332 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:45.925344 (+ 12us) server_negotiation.cc:408] Connection header received
1025 08:02:45.925385 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:45.925387 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:45.925427 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:45.925510 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:45.925805 (+ 295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.926510 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.927371 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.928006 (+ 635us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.928050 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:45.928095 (+ 45us) server_negotiation.cc:300] Negotiation successful
1025 08:02:45.928169 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20251025 08:02:45.936669 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.930002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.27.132.60:37999)
1025 08:02:45.930109 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:45.930113 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:45.930126 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:02:45.930221 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:45.930224 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:45.930264 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:45.930338 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:45.931030 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.931792 (+ 762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.932404 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.932606 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.933717 (+ 1111us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:45.933733 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:45.933736 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:45.933761 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:45.935405 (+ 1644us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:45.935788 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:45.935790 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:45.935792 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:45.935833 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:45.935989 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:45.935991 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:45.935992 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:45.936116 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:45.936215 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:45.936470 (+ 255us) server_negotiation.cc:300] Negotiation successful
1025 08:02:45.936564 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 08:02:45.939795 3328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.937427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:35032)
1025 08:02:45.937650 (+ 223us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:45.937662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:45.937743 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:45.937956 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:45.937958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:45.937968 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:45.938139 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.938144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.938858 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.938860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:45.939509 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.939517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.939605 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.939619 (+ 14us) client_negotiation.cc:769] Sending connection context
1025 08:02:45.939655 (+ 36us) client_negotiation.cc:240] Negotiation successful
1025 08:02:45.939701 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20251025 08:02:45.940259 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.937560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.27.132.62:42629)
1025 08:02:45.937676 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:45.937679 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:45.937709 (+ 30us) server_negotiation.cc:408] Connection header received
1025 08:02:45.937814 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:45.937819 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:45.937881 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:45.937946 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:45.938238 (+ 292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.938753 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.939624 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.940040 (+ 416us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.940082 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:45.940124 (+ 42us) server_negotiation.cc:300] Negotiation successful
1025 08:02:45.940172 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251025 08:02:45.943598 3328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.941459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.61:34389 (local address 127.0.0.1:52476)
1025 08:02:45.941576 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:45.941592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:45.941666 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:45.941871 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:45.941873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:45.941884 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:45.942017 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.942021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.942686 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.942689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:45.943220 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.943226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.943360 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.943378 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 08:02:45.943430 (+ 52us) client_negotiation.cc:240] Negotiation successful
1025 08:02:45.943494 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20251025 08:02:45.944057 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.941557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.27.132.61:34389)
1025 08:02:45.941689 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:45.941693 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:45.941721 (+ 28us) server_negotiation.cc:408] Connection header received
1025 08:02:45.941763 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:45.941765 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:45.941804 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:45.941862 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:45.942132 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.942598 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.943363 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.943858 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.943888 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:45.943929 (+ 41us) server_negotiation.cc:300] Negotiation successful
1025 08:02:45.943975 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 08:02:45.947279 3328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.945159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.60:37999 (local address 127.0.0.1:37888)
1025 08:02:45.945285 (+ 126us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:45.945301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:45.945375 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:45.945589 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:45.945593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:45.945616 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:45.945771 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.945776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.946413 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.946416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:45.946947 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:45.946954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.947055 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.947070 (+ 15us) client_negotiation.cc:769] Sending connection context
1025 08:02:45.947120 (+ 50us) client_negotiation.cc:240] Negotiation successful
1025 08:02:45.947169 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"spinlock_wait_cycles":1280}
I20251025 08:02:45.947712 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:45.945257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.27.132.60:37999)
1025 08:02:45.945356 (+ 99us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:45.945360 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:45.945371 (+ 11us) server_negotiation.cc:408] Connection header received
1025 08:02:45.945457 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:45.945463 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:45.945500 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:45.945564 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:45.945868 (+ 304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.946305 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:45.947052 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:45.947531 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:45.947557 (+ 26us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:45.947605 (+ 48us) server_negotiation.cc:300] Negotiation successful
1025 08:02:45.947643 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27,"spinlock_wait_cycles":3328}
I20251025 08:02:45.949072 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 2534
I20251025 08:02:45.955204 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:46805
--local_ip_for_outbound_sockets=127.27.132.1
--tserver_master_addrs=127.27.132.62:42629
--webserver_port=40489
--webserver_interface=127.27.132.1
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.132.62:42629,127.27.132.61:34389
--tserver_master_addrs=127.27.132.62:42629,127.27.132.61:34389,127.27.132.60:37999 with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:46.068634 3329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:46.068959 3329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:46.069043 3329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:46.072827 3329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:46.072901 3329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:46.072984 3329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:02:46.077452 3329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:46805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=40489
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:42629,127.27.132.61:34389,127.27.132.60:37999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:46.078481 3329 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:46.079285 3329 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:46.085973 3335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:46.085965 3334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:46.085978 3337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:46.086344 3329 server_base.cc:1047] running on GCE node
I20251025 08:02:46.086709 3329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:46.087214 3329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:46.088367 3329 hybrid_clock.cc:648] HybridClock initialized: now 1761379366088347 us; error 36 us; skew 500 ppm
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:46.091264 3329 init.cc:377] Logged in from keytab as oryx/127.27.132.1@KRBTEST.COM (short username oryx)
I20251025 08:02:46.092378 3329 webserver.cc:492] Webserver started at http://127.27.132.1:40489/ using document root <none> and password file <none>
I20251025 08:02:46.092931 3329 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:46.092993 3329 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:46.096539 3329 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:02:46.098318 3344 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:46.099303 3329 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:46.099462 3329 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "9e8418b907064b84a8752c25c6fd5890"
format_stamp: "Formatted at 2025-10-25 08:02:43 on dist-test-slave-lq8l"
server_key: "162d083991e712b773d6f2bea60216f0"
server_key_iv: "ebcdb9cd1168d451ec7a74cbda39fe0d"
server_key_version: "encryptionkey@0"
I20251025 08:02:46.099853 3329 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:46.117786 3329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:46.120957 3329 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:46.121119 3329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:46.121639 3329 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:46.122532 3329 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:46.122578 3329 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:46.122620 3329 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:46.122641 3329 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:46.132239 3329 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:46805
I20251025 08:02:46.132257 3457 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:46805 every 8 connection(s)
I20251025 08:02:46.133256 3329 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:02:46.141232 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 3329
I20251025 08:02:46.141448 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 2670
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.60@KRBTEST.COM
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.61@KRBTEST.COM
I20251025 08:02:46.150135 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:41457
--local_ip_for_outbound_sockets=127.27.132.2
--tserver_master_addrs=127.27.132.62:42629
--webserver_port=33197
--webserver_interface=127.27.132.2
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.132.62:42629,127.27.132.61:34389
--tserver_master_addrs=127.27.132.62:42629,127.27.132.61:34389,127.27.132.60:37999 with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251025 08:02:46.150218 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.136469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:37095 (local address 127.27.132.61:34389)
1025 08:02:46.136602 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.136607 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.136727 (+ 120us) server_negotiation.cc:408] Connection header received
1025 08:02:46.137374 (+ 647us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.137378 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.137432 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.137517 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.139060 (+ 1543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.139609 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.140531 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.140730 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.145046 (+ 4316us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.145067 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.145070 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.145100 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.147617 (+ 2517us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.148504 (+ 887us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.148508 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.148510 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.148563 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.148898 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.148902 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.148904 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.149104 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.149168 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.149909 (+ 741us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.150048 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251025 08:02:46.150218 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.135039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:57189 (local address 127.27.132.60:37999)
1025 08:02:46.135238 (+ 199us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.135243 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.136090 (+ 847us) server_negotiation.cc:408] Connection header received
1025 08:02:46.137221 (+ 1131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.137227 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.137287 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.137359 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.139083 (+ 1724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.139867 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.140662 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.140868 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.145055 (+ 4187us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.145073 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.145076 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.145106 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.147955 (+ 2849us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.148509 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.148513 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.148514 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.148563 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.148898 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.148902 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.148904 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.149094 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.149168 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.149909 (+ 741us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.150048 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20251025 08:02:46.150238 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.135066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:57781 (local address 127.27.132.62:42629)
1025 08:02:46.135230 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.135235 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.136355 (+ 1120us) server_negotiation.cc:408] Connection header received
1025 08:02:46.137104 (+ 749us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.137108 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.137175 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.137267 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.139004 (+ 1737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.139853 (+ 849us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.141014 (+ 1161us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.141154 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.144914 (+ 3760us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.144942 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.144949 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.144986 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.147700 (+ 2714us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.148458 (+ 758us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.148468 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.148473 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.148534 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.148891 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.148898 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.148902 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.149112 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.149206 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.149920 (+ 714us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.150069 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251025 08:02:46.151731 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.135422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.1:57781)
1025 08:02:46.136256 (+ 834us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.136271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.136892 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.137479 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.137491 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.138337 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:46.138847 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.138860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.139960 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.139964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.140493 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.140504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.140696 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.142084 (+ 1388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:46.142124 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:46.144711 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:46.147837 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.147843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.147848 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.148316 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.148640 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.148646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.148651 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.148782 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.149261 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:46.149267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:46.149602 (+ 335us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.150414 (+ 812us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.150766 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":73,"spinlock_wait_cycles":33536,"thread_start_us":95,"threads_started":1}
I20251025 08:02:46.151733 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.135422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.60:37999 (local address 127.27.132.1:57189)
1025 08:02:46.135888 (+ 466us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.135932 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.136872 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.137478 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.137489 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.138191 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:46.138904 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.138914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.140040 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.140043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.140424 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.140430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.140698 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.142227 (+ 1529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:46.142239 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:46.144876 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:46.148079 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.148084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.148086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.148384 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.148679 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.148682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.148684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.148788 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.149258 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:46.149267 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:46.149585 (+ 318us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.150415 (+ 830us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.150873 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":141,"spinlock_wait_cycles":25984,"thread_start_us":113,"threads_started":1}
I20251025 08:02:46.151788 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.136326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.61:34389 (local address 127.27.132.1:37095)
1025 08:02:46.136639 (+ 313us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.136652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.136887 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.137539 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.137542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.138191 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:46.138904 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.138914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.139744 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.139747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.140351 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.140360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.140849 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.142084 (+ 1235us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:46.142124 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:46.144876 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:46.147777 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.147784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.147800 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.148361 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.148679 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.148682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.148684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.148788 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.149258 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:46.149267 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:46.149585 (+ 318us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.151487 (+ 1902us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.151629 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"spinlock_wait_cycles":47232,"thread_start_us":77,"threads_started":1}
I20251025 08:02:46.153015 3458 heartbeater.cc:344] Connected to a master server at 127.27.132.60:37999
I20251025 08:02:46.153285 3458 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:46.153486 3459 heartbeater.cc:344] Connected to a master server at 127.27.132.62:42629
I20251025 08:02:46.153578 3459 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:46.153893 3458 heartbeater.cc:507] Master 127.27.132.60:37999 requested a full tablet report, sending...
I20251025 08:02:46.153891 3459 heartbeater.cc:507] Master 127.27.132.62:42629 requested a full tablet report, sending...
I20251025 08:02:46.154816 3460 heartbeater.cc:344] Connected to a master server at 127.27.132.61:34389
I20251025 08:02:46.154899 3460 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:46.155117 3460 heartbeater.cc:507] Master 127.27.132.61:34389 requested a full tablet report, sending...
I20251025 08:02:46.155289 2470 ts_manager.cc:194] Re-registered known tserver with Master: 9e8418b907064b84a8752c25c6fd5890 (127.27.132.1:46805)
I20251025 08:02:46.155490 3266 ts_manager.cc:194] Registered new tserver with Master: 9e8418b907064b84a8752c25c6fd5890 (127.27.132.1:46805)
I20251025 08:02:46.156024 2470 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:57781
I20251025 08:02:46.157344 3082 ts_manager.cc:194] Registered new tserver with Master: 9e8418b907064b84a8752c25c6fd5890 (127.27.132.1:46805)
W20251025 08:02:46.259471 3465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:46.259696 3465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:46.259755 3465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:46.262869 3465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:46.262923 3465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:46.263000 3465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:02:46.267009 3465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:41457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=33197
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:42629,127.27.132.61:34389,127.27.132.60:37999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:46.268018 3465 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:46.268805 3465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:46.275095 3470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:46.275105 3471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:46.275247 3473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:46.276075 3465 server_base.cc:1047] running on GCE node
I20251025 08:02:46.276396 3465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:46.276897 3465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:46.278046 3465 hybrid_clock.cc:648] HybridClock initialized: now 1761379366278021 us; error 42 us; skew 500 ppm
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:46.280869 3465 init.cc:377] Logged in from keytab as oryx/127.27.132.2@KRBTEST.COM (short username oryx)
I20251025 08:02:46.281841 3465 webserver.cc:492] Webserver started at http://127.27.132.2:33197/ using document root <none> and password file <none>
I20251025 08:02:46.282348 3465 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:46.282404 3465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:46.285816 3465 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:46.287678 3480 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:46.288651 3465 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:46.288756 3465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "c5f1b63c203848ba95feb28f30b6b891"
format_stamp: "Formatted at 2025-10-25 08:02:43 on dist-test-slave-lq8l"
server_key: "ced925c934a0f3bfe5c178a486814c3a"
server_key_iv: "994d565ff359ce2040e0e41d9b8ec497"
server_key_version: "encryptionkey@0"
I20251025 08:02:46.289137 3465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:46.310676 3465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:46.313731 3465 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:46.313905 3465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:46.314445 3465 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:46.315296 3465 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:46.315390 3465 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:46.315452 3465 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:46.315485 3465 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:46.324347 3465 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:41457
I20251025 08:02:46.324379 3593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:41457 every 8 connection(s)
I20251025 08:02:46.325330 3465 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:02:46.326663 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 3465
I20251025 08:02:46.326850 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 2806
I20251025 08:02:46.334779 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:42201
--local_ip_for_outbound_sockets=127.27.132.3
--tserver_master_addrs=127.27.132.62:42629
--webserver_port=45259
--webserver_interface=127.27.132.3
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.132.62:42629,127.27.132.61:34389
--tserver_master_addrs=127.27.132.62:42629,127.27.132.61:34389,127.27.132.60:37999 with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.60@KRBTEST.COM
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.61@KRBTEST.COM
I20251025 08:02:46.342144 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.329621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:33043 (local address 127.27.132.60:37999)
1025 08:02:46.329771 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.329776 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.330005 (+ 229us) server_negotiation.cc:408] Connection header received
1025 08:02:46.330167 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.330170 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.330231 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.330287 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.331797 (+ 1510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.332575 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.333520 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.333729 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.337417 (+ 3688us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.337446 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.337452 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.337483 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.339679 (+ 2196us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.340360 (+ 681us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.340365 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.340369 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.340421 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.340829 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.340834 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.340839 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.341044 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.341115 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.341851 (+ 736us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.341975 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251025 08:02:46.342950 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.329279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.60:37999 (local address 127.27.132.2:33043)
1025 08:02:46.329935 (+ 656us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.329947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.330062 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.330441 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.330444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.330788 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:46.331622 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.331638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.332729 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.332733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.333386 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.333391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.333925 (+ 534us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.335050 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:46.335071 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:46.337232 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:46.339824 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.339833 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.339849 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.340239 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.340540 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.340544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.340546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.340679 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.341195 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:46.341204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:46.341514 (+ 310us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.342574 (+ 1060us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.342717 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"spinlock_wait_cycles":96640,"thread_start_us":75,"threads_started":1}
I20251025 08:02:46.343025 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.327477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:50925 (local address 127.27.132.62:42629)
1025 08:02:46.327799 (+ 322us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.327805 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.328213 (+ 408us) server_negotiation.cc:408] Connection header received
1025 08:02:46.329497 (+ 1284us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.329504 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.329571 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.329629 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.331802 (+ 2173us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.332649 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.333606 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.333832 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.338718 (+ 4886us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.338746 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.338751 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.338791 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.340919 (+ 2128us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.341502 (+ 583us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.341507 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.341510 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.341572 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.341916 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.341923 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.341928 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.342141 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.342235 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.342638 (+ 403us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.342788 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20251025 08:02:46.343533 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.327619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:43517 (local address 127.27.132.61:34389)
1025 08:02:46.327903 (+ 284us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.327908 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.329280 (+ 1372us) server_negotiation.cc:408] Connection header received
1025 08:02:46.329506 (+ 226us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.329510 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.329569 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.329636 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.331797 (+ 2161us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.332581 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.333520 (+ 939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.333711 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.337939 (+ 4228us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.337963 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.337970 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.338006 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.340429 (+ 2423us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.340943 (+ 514us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.340946 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.340948 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.340992 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.341261 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.341264 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.341266 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.341430 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.341517 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.343084 (+ 1567us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.343225 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20251025 08:02:46.342959 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.327408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.61:34389 (local address 127.27.132.2:43517)
1025 08:02:46.329187 (+ 1779us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.329203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.329357 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.330039 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.330049 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.330788 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:46.331623 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.331638 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.332729 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.332733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.333383 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.333391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.333925 (+ 534us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.335143 (+ 1218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:46.335161 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:46.337789 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:46.340591 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.340594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.340596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.340816 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.341101 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.341104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.341106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.341165 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.341558 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:46.341561 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:46.341638 (+ 77us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.341783 (+ 145us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.342068 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1657,"mutex_wait_us":1448,"spinlock_wait_cycles":6656,"thread_start_us":97,"threads_started":1}
I20251025 08:02:46.343689 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.327408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.2:50925)
1025 08:02:46.328077 (+ 669us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.328123 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.329237 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.330040 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.330051 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.330786 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:46.331624 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.331641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.332797 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.332803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.333463 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.333475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.333960 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.334779 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:46.334803 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:46.338548 (+ 3745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:46.341101 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.341107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.341112 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.341353 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.341724 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.341730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.341734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.341799 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.342305 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:46.342308 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:46.342405 (+ 97us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.343235 (+ 830us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.343458 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"spinlock_wait_cycles":18176,"thread_start_us":141,"threads_started":1}
I20251025 08:02:46.344825 3596 heartbeater.cc:344] Connected to a master server at 127.27.132.61:34389
I20251025 08:02:46.344897 3595 heartbeater.cc:344] Connected to a master server at 127.27.132.62:42629
I20251025 08:02:46.345073 3595 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:46.345073 3596 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:46.345098 3594 heartbeater.cc:344] Connected to a master server at 127.27.132.60:37999
I20251025 08:02:46.345747 3596 heartbeater.cc:507] Master 127.27.132.61:34389 requested a full tablet report, sending...
I20251025 08:02:46.345896 3594 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:46.346081 3594 heartbeater.cc:507] Master 127.27.132.60:37999 requested a full tablet report, sending...
I20251025 08:02:46.345747 3595 heartbeater.cc:507] Master 127.27.132.62:42629 requested a full tablet report, sending...
I20251025 08:02:46.346978 3082 ts_manager.cc:194] Registered new tserver with Master: c5f1b63c203848ba95feb28f30b6b891 (127.27.132.2:41457)
I20251025 08:02:46.347007 3266 ts_manager.cc:194] Registered new tserver with Master: c5f1b63c203848ba95feb28f30b6b891 (127.27.132.2:41457)
I20251025 08:02:46.347674 2470 ts_manager.cc:194] Re-registered known tserver with Master: c5f1b63c203848ba95feb28f30b6b891 (127.27.132.2:41457)
I20251025 08:02:46.348261 2470 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:50925
W20251025 08:02:46.444433 3602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:46.444680 3602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:46.444747 3602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:46.448089 3602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:46.448148 3602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:46.448256 3602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:02:46.452535 3602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:42201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=45259
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:42629,127.27.132.61:34389,127.27.132.60:37999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:46.453543 3602 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:46.454365 3602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:46.460609 3610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:46.460597 3607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:46.460600 3608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:46.460688 3602 server_base.cc:1047] running on GCE node
I20251025 08:02:46.461136 3602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:46.461645 3602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:46.462796 3602 hybrid_clock.cc:648] HybridClock initialized: now 1761379366462788 us; error 31 us; skew 500 ppm
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:46.465586 3602 init.cc:377] Logged in from keytab as oryx/127.27.132.3@KRBTEST.COM (short username oryx)
I20251025 08:02:46.466646 3602 webserver.cc:492] Webserver started at http://127.27.132.3:45259/ using document root <none> and password file <none>
I20251025 08:02:46.467187 3602 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:46.467259 3602 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:46.470662 3602 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20251025 08:02:46.472566 3617 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:46.473518 3602 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:46.473655 3602 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "384defb513fd42959725a27971ee3759"
format_stamp: "Formatted at 2025-10-25 08:02:44 on dist-test-slave-lq8l"
server_key: "fb2dbbe5e5f56feea8a94e0a2622d16e"
server_key_iv: "0074d8b08c5815d1997182c40dc67a9f"
server_key_version: "encryptionkey@0"
I20251025 08:02:46.474156 3602 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:46.490937 3602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:46.493744 3602 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:46.493893 3602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:46.494359 3602 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:02:46.495204 3602 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:46.495251 3602 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:46.495290 3602 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:46.495312 3602 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:46.504321 3602 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:42201
I20251025 08:02:46.504369 3730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:42201 every 8 connection(s)
I20251025 08:02:46.505358 3602 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:02:46.511415 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 3602
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.61@KRBTEST.COM
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.60@KRBTEST.COM
Oct 25 08:02:46 dist-test-slave-lq8l krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379366, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:46.521719 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.508397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:48183 (local address 127.27.132.60:37999)
1025 08:02:46.508538 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.508544 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.508593 (+ 49us) server_negotiation.cc:408] Connection header received
1025 08:02:46.509224 (+ 631us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.509231 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.509286 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.509366 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.510877 (+ 1511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.511643 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.513044 (+ 1401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.513255 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.517374 (+ 4119us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.517392 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.517395 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.517421 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.519209 (+ 1788us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.519888 (+ 679us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.519891 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.519894 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.519947 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.520326 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.520333 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.520336 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.520531 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.520606 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.521366 (+ 760us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.521484 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251025 08:02:46.522722 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.507128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:49439 (local address 127.27.132.61:34389)
1025 08:02:46.507266 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.507270 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.508299 (+ 1029us) server_negotiation.cc:408] Connection header received
1025 08:02:46.509418 (+ 1119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.509422 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.509489 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.509563 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.510787 (+ 1224us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.511602 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.512528 (+ 926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.512725 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.517830 (+ 5105us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.517854 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.517858 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.517895 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.520132 (+ 2237us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.521498 (+ 1366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.521502 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.521504 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.521554 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.521855 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.521860 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.521863 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.522019 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.522093 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.522425 (+ 332us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.522543 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251025 08:02:46.522950 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.507157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:49353 (local address 127.27.132.62:42629)
1025 08:02:46.507294 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.507298 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.508053 (+ 755us) server_negotiation.cc:408] Connection header received
1025 08:02:46.509116 (+ 1063us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.509119 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.509187 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.509262 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.511794 (+ 2532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.512385 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.513939 (+ 1554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.514145 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.517060 (+ 2915us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.517080 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.517083 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.517112 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.520888 (+ 3776us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.521503 (+ 615us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.521507 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.521509 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.521557 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.522094 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.522098 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.522101 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.522262 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.522337 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.522678 (+ 341us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.522794 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251025 08:02:46.523276 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.508267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.60:37999 (local address 127.27.132.3:48183)
1025 08:02:46.508522 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.508536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.508920 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.509436 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.509448 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.510027 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:46.510695 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.510704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.512540 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.512543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.512931 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.512938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.513048 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.514529 (+ 1481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:46.514546 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:46.517269 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:46.519367 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.519377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.519391 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.519762 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.520065 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.520071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.520074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.520206 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.520712 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:46.520721 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:46.521051 (+ 330us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.521965 (+ 914us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.522293 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"spinlock_wait_cycles":18176,"thread_start_us":83,"threads_started":1}
I20251025 08:02:46.523319 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.507441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.61:34389 (local address 127.27.132.3:49439)
1025 08:02:46.508206 (+ 765us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.508222 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.508920 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.509599 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.509603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.510010 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:46.510632 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.510645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.511740 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.511745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.512376 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.512390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.512954 (+ 564us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.513844 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:46.513878 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:46.517670 (+ 3792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:46.521037 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.521040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.521041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.521375 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.521668 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.521672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.521674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.521745 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.522163 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:46.522166 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:46.522244 (+ 78us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.522909 (+ 665us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.523121 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":23,"spinlock_wait_cycles":71808,"thread_start_us":149,"threads_started":1}
I20251025 08:02:46.524120 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.507443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.3:49353)
1025 08:02:46.507891 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.507932 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.508905 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.509447 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.509450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.510430 (+ 980us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:46.510706 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.510712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.513271 (+ 2559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.513275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.513801 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.513813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.514271 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.515117 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:46.515133 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:46.516907 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:46.520995 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.521002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.521005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.521374 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.521668 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:46.521672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:46.521674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:46.521745 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:46.522404 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:46.522407 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:46.522488 (+ 81us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.523819 (+ 1331us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.523953 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"spinlock_wait_cycles":26368,"thread_start_us":116,"threads_started":1}
I20251025 08:02:46.524633 3731 heartbeater.cc:344] Connected to a master server at 127.27.132.60:37999
I20251025 08:02:46.524786 3733 heartbeater.cc:344] Connected to a master server at 127.27.132.61:34389
I20251025 08:02:46.524977 3733 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:46.524977 3731 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:46.525655 3731 heartbeater.cc:507] Master 127.27.132.60:37999 requested a full tablet report, sending...
I20251025 08:02:46.525986 3732 heartbeater.cc:344] Connected to a master server at 127.27.132.62:42629
I20251025 08:02:46.526067 3732 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:46.526244 3732 heartbeater.cc:507] Master 127.27.132.62:42629 requested a full tablet report, sending...
I20251025 08:02:46.526968 2470 ts_manager.cc:194] Re-registered known tserver with Master: 384defb513fd42959725a27971ee3759 (127.27.132.3:42201)
I20251025 08:02:46.527238 3266 ts_manager.cc:194] Registered new tserver with Master: 384defb513fd42959725a27971ee3759 (127.27.132.3:42201)
I20251025 08:02:46.525655 3733 heartbeater.cc:507] Master 127.27.132.61:34389 requested a full tablet report, sending...
I20251025 08:02:46.527693 2470 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:49353
I20251025 08:02:46.528241 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.514405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.27.132.60:37999)
1025 08:02:46.514896 (+ 491us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.514900 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.515877 (+ 977us) server_negotiation.cc:408] Connection header received
1025 08:02:46.515916 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.515919 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.515973 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.516052 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.518416 (+ 2364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.519264 (+ 848us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.520477 (+ 1213us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.520682 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.522820 (+ 2138us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.522837 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.522840 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.522864 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.526024 (+ 3160us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.526425 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.526428 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.526430 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.526489 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.526802 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.526807 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.526809 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.527011 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.527204 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.527889 (+ 685us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.528052 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":62,"threads_started":1}
I20251025 08:02:46.528370 3082 ts_manager.cc:194] Registered new tserver with Master: 384defb513fd42959725a27971ee3759 (127.27.132.3:42201)
I20251025 08:02:46.530802 3747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.514562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.27.132.61:34389)
1025 08:02:46.515196 (+ 634us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.515200 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.515214 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:02:46.515254 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.515257 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.515305 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.515425 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.516594 (+ 1169us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.517662 (+ 1068us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.524985 (+ 7323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.525241 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.527301 (+ 2060us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.527320 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.527359 (+ 39us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.527388 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.529224 (+ 1836us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.529675 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.529679 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.529680 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.529729 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.529969 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.529972 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.529974 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.530138 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.530279 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.530500 (+ 221us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.530630 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":68,"threads_started":1}
I20251025 08:02:46.531211 3743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.513618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.27.132.62:42629)
1025 08:02:46.515796 (+ 2178us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.515800 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.515814 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:02:46.515853 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.515856 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.515909 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.516978 (+ 1069us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.517080 (+ 102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.517818 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.521066 (+ 3248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.521260 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.521365 (+ 105us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.521384 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.521387 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.521405 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.529550 (+ 8145us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.529996 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.530001 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.530003 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.530059 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.530315 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.530318 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.530320 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.530500 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.530638 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.530925 (+ 287us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.531063 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2090,"thread_start_us":76,"threads_started":1}
I20251025 08:02:46.533522 2470 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:02:46.535764 2470 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:02:46.552564 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.548278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:42201 (local address 127.0.0.1:56892)
1025 08:02:46.548534 (+ 256us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.548549 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.548643 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.549950 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.549954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.549993 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:46.550261 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.550272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.551541 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.551545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.552151 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.552158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.552275 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.552299 (+ 24us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.552352 (+ 53us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.552413 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"spinlock_wait_cycles":6784,"thread_start_us":103,"threads_started":1}
I20251025 08:02:46.552801 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.548923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:41457 (local address 127.0.0.1:56072)
1025 08:02:46.549152 (+ 229us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.549164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.549261 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.549811 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.549814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.549831 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:46.550080 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.550086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.551506 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.551510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.552433 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.552443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.552534 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.552550 (+ 16us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.552594 (+ 44us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.552649 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":103,"threads_started":1}
I20251025 08:02:46.553071 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.548595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:46805 (local address 127.0.0.1:36526)
1025 08:02:46.548829 (+ 234us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.548841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.548919 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.549950 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.549954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.549994 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:46.550287 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.550295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.551636 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.551639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.552460 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.552467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.552532 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.552553 (+ 21us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.552594 (+ 41us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.552649 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"spinlock_wait_cycles":9856,"thread_start_us":110,"threads_started":1}
I20251025 08:02:46.553359 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.548813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.27.132.3:42201)
1025 08:02:46.549562 (+ 749us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.549568 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.549586 (+ 18us) server_negotiation.cc:408] Connection header received
1025 08:02:46.549650 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.549654 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.549789 (+ 135us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.550425 (+ 636us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:46.550579 (+ 154us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.551423 (+ 844us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.552301 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.552953 (+ 652us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.553084 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.553158 (+ 74us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.553248 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":95,"threads_started":1}
I20251025 08:02:46.553718 3761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.549147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.27.132.1:46805)
1025 08:02:46.549490 (+ 343us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.549499 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.549520 (+ 21us) server_negotiation.cc:408] Connection header received
1025 08:02:46.549606 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.549613 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.549790 (+ 177us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.550150 (+ 360us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:46.550438 (+ 288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.551529 (+ 1091us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.552775 (+ 1246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.553324 (+ 549us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.553441 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.553500 (+ 59us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.553585 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":113,"threads_started":1}
I20251025 08:02:46.554270 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.549076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.27.132.2:41457)
1025 08:02:46.549406 (+ 330us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.549412 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.549442 (+ 30us) server_negotiation.cc:408] Connection header received
1025 08:02:46.549517 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.549521 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.549671 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.549812 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:46.550217 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.551372 (+ 1155us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.553408 (+ 2036us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.553931 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.554017 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.554073 (+ 56us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.554145 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":121,"threads_started":1}
I20251025 08:02:46.556084 3392 tablet_service.cc:1505] Processing CreateTablet for tablet 67a14d9714eb4c158aaf7a9997799942 (DEFAULT_TABLE table=test-table [id=063f9c5854974dfdb774035d0bbd3eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:46.556084 3665 tablet_service.cc:1505] Processing CreateTablet for tablet 67a14d9714eb4c158aaf7a9997799942 (DEFAULT_TABLE table=test-table [id=063f9c5854974dfdb774035d0bbd3eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:46.556880 3392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67a14d9714eb4c158aaf7a9997799942. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:46.556939 3665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67a14d9714eb4c158aaf7a9997799942. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:46.557782 3526 tablet_service.cc:1505] Processing CreateTablet for tablet 67a14d9714eb4c158aaf7a9997799942 (DEFAULT_TABLE table=test-table [id=063f9c5854974dfdb774035d0bbd3eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:46.558600 3526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67a14d9714eb4c158aaf7a9997799942. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:46.562549 3762 tablet_bootstrap.cc:492] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: Bootstrap starting.
I20251025 08:02:46.563697 3763 tablet_bootstrap.cc:492] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759: Bootstrap starting.
I20251025 08:02:46.565012 3762 tablet_bootstrap.cc:654] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:46.565862 3762 log.cc:826] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:46.566076 3763 tablet_bootstrap.cc:654] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:46.566087 3764 tablet_bootstrap.cc:492] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: Bootstrap starting.
I20251025 08:02:46.566895 3763 log.cc:826] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:46.568176 3762 tablet_bootstrap.cc:492] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: No bootstrap required, opened a new log
I20251025 08:02:46.568192 3764 tablet_bootstrap.cc:654] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:46.568387 3762 ts_tablet_manager.cc:1403] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251025 08:02:46.568439 3763 tablet_bootstrap.cc:492] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759: No bootstrap required, opened a new log
I20251025 08:02:46.568599 3763 ts_tablet_manager.cc:1403] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 08:02:46.568949 3764 log.cc:826] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:46.570930 3764 tablet_bootstrap.cc:492] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: No bootstrap required, opened a new log
I20251025 08:02:46.571116 3764 ts_tablet_manager.cc:1403] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 08:02:46.571470 3763 raft_consensus.cc:359] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:46.571484 3762 raft_consensus.cc:359] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:46.571652 3763 raft_consensus.cc:385] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:46.571676 3762 raft_consensus.cc:385] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:46.571718 3763 raft_consensus.cc:740] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384defb513fd42959725a27971ee3759, State: Initialized, Role: FOLLOWER
I20251025 08:02:46.571733 3762 raft_consensus.cc:740] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8418b907064b84a8752c25c6fd5890, State: Initialized, Role: FOLLOWER
I20251025 08:02:46.572151 3763 consensus_queue.cc:260] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:46.572160 3762 consensus_queue.cc:260] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:46.572997 3763 ts_tablet_manager.cc:1434] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251025 08:02:46.573001 3762 ts_tablet_manager.cc:1434] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251025 08:02:46.573271 3732 heartbeater.cc:499] Master 127.27.132.62:42629 was elected leader, sending a full tablet report...
I20251025 08:02:46.573527 3459 heartbeater.cc:499] Master 127.27.132.62:42629 was elected leader, sending a full tablet report...
I20251025 08:02:46.574229 3764 raft_consensus.cc:359] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:46.574427 3764 raft_consensus.cc:385] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:46.574489 3764 raft_consensus.cc:740] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f1b63c203848ba95feb28f30b6b891, State: Initialized, Role: FOLLOWER
I20251025 08:02:46.574895 3764 consensus_queue.cc:260] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:46.575762 3764 ts_tablet_manager.cc:1434] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251025 08:02:46.576617 3595 heartbeater.cc:499] Master 127.27.132.62:42629 was elected leader, sending a full tablet report...
W20251025 08:02:46.576670 3597 tablet.cc:2378] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 08:02:46.628505 3768 raft_consensus.cc:493] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:02:46.628691 3768 raft_consensus.cc:515] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:46.629817 3768 leader_election.cc:290] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e8418b907064b84a8752c25c6fd5890 (127.27.132.1:46805), c5f1b63c203848ba95feb28f30b6b891 (127.27.132.2:41457)
I20251025 08:02:46.630465 3769 raft_consensus.cc:493] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:02:46.630628 3769 raft_consensus.cc:515] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:46.633459 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.630091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:46805 (local address 127.27.132.3:39383)
1025 08:02:46.630233 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.630252 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.630361 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.630828 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.630832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.630849 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:46.631105 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.631112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.632141 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.632145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.633071 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.633080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.633180 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.633201 (+ 21us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.633246 (+ 45us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.633299 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20251025 08:02:46.633481 3769 leader_election.cc:290] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 384defb513fd42959725a27971ee3759 (127.27.132.3:42201), c5f1b63c203848ba95feb28f30b6b891 (127.27.132.2:41457)
I20251025 08:02:46.634371 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.630321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:41457 (local address 127.27.132.3:33911)
1025 08:02:46.630605 (+ 284us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.630620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.630694 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.631422 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.631425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.631451 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:46.631691 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.631700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.632797 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.632800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.634023 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.634034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.634130 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.634148 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.634196 (+ 48us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.634256 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20251025 08:02:46.634583 3761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.630169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:39383 (local address 127.27.132.1:46805)
1025 08:02:46.630512 (+ 343us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.630517 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.630533 (+ 16us) server_negotiation.cc:408] Connection header received
1025 08:02:46.630585 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.630589 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.630641 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.630719 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:46.631230 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.632020 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.633840 (+ 1820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.634328 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.634366 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.634411 (+ 45us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.634472 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20251025 08:02:46.635250 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.632075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:42201 (local address 127.27.132.1:55425)
1025 08:02:46.632300 (+ 225us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.632313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.632385 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.632879 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.632883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.632901 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:46.633140 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.633147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.634055 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.634058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.634888 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.634898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.635006 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.635024 (+ 18us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.635070 (+ 46us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.635156 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20251025 08:02:46.635318 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.630928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:33911 (local address 127.27.132.2:41457)
1025 08:02:46.631054 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.631058 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.631072 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:02:46.631123 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.631126 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.631176 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.631250 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:46.631845 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.632668 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.634605 (+ 1937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.635081 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.635115 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.635160 (+ 45us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.635210 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251025 08:02:46.635345 3411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67a14d9714eb4c158aaf7a9997799942" candidate_uuid: "384defb513fd42959725a27971ee3759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8418b907064b84a8752c25c6fd5890" is_pre_election: true
W20251025 08:02:46.635484 3461 tablet.cc:2378] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 08:02:46.635545 3411 raft_consensus.cc:2468] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 384defb513fd42959725a27971ee3759 in term 0.
I20251025 08:02:46.636322 3620 leader_election.cc:304] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384defb513fd42959725a27971ee3759, 9e8418b907064b84a8752c25c6fd5890; no voters:
I20251025 08:02:46.636545 3548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67a14d9714eb4c158aaf7a9997799942" candidate_uuid: "384defb513fd42959725a27971ee3759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f1b63c203848ba95feb28f30b6b891" is_pre_election: true
I20251025 08:02:46.636615 3768 raft_consensus.cc:2804] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:02:46.636786 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.632509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:55425 (local address 127.27.132.3:42201)
1025 08:02:46.632613 (+ 104us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.632617 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.632628 (+ 11us) server_negotiation.cc:408] Connection header received
1025 08:02:46.632664 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.632666 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.632723 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.632792 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:46.633268 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.633931 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.636039 (+ 2108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.636514 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.636547 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.636593 (+ 46us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.636651 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 08:02:46.636770 3548 raft_consensus.cc:2468] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 384defb513fd42959725a27971ee3759 in term 0.
I20251025 08:02:46.637209 3685 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67a14d9714eb4c158aaf7a9997799942" candidate_uuid: "9e8418b907064b84a8752c25c6fd5890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384defb513fd42959725a27971ee3759" is_pre_election: true
I20251025 08:02:46.636683 3768 raft_consensus.cc:493] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:02:46.637315 3768 raft_consensus.cc:3060] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:46.638572 3768 raft_consensus.cc:515] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:46.638840 3685 raft_consensus.cc:2393] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e8418b907064b84a8752c25c6fd5890 in current term 1: Already voted for candidate 384defb513fd42959725a27971ee3759 in this term.
I20251025 08:02:46.639146 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.633696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:41457 (local address 127.27.132.1:33573)
1025 08:02:46.633921 (+ 225us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:46.633933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:46.634002 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:46.635768 (+ 1766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:46.635772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:46.635788 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:46.636002 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.636008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.637886 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.637889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:46.638757 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:46.638768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.638885 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.638904 (+ 19us) client_negotiation.cc:769] Sending connection context
1025 08:02:46.638956 (+ 52us) client_negotiation.cc:240] Negotiation successful
1025 08:02:46.639016 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20251025 08:02:46.639869 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.634153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:33573 (local address 127.27.132.2:41457)
1025 08:02:46.635509 (+ 1356us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.635513 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.635525 (+ 12us) server_negotiation.cc:408] Connection header received
1025 08:02:46.635561 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.635564 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.635609 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.635683 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:46.637049 (+ 1366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.637750 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.638905 (+ 1155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.639619 (+ 714us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.639654 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.639713 (+ 59us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.639761 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1296,"thread_start_us":117,"threads_started":1}
I20251025 08:02:46.640079 3768 leader_election.cc:290] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [CANDIDATE]: Term 1 election: Requested vote from peers 9e8418b907064b84a8752c25c6fd5890 (127.27.132.1:46805), c5f1b63c203848ba95feb28f30b6b891 (127.27.132.2:41457)
I20251025 08:02:46.640106 3411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67a14d9714eb4c158aaf7a9997799942" candidate_uuid: "384defb513fd42959725a27971ee3759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8418b907064b84a8752c25c6fd5890"
I20251025 08:02:46.640231 3411 raft_consensus.cc:3060] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:46.640328 3548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67a14d9714eb4c158aaf7a9997799942" candidate_uuid: "384defb513fd42959725a27971ee3759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f1b63c203848ba95feb28f30b6b891"
I20251025 08:02:46.640439 3548 raft_consensus.cc:3060] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:46.640520 3547 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67a14d9714eb4c158aaf7a9997799942" candidate_uuid: "9e8418b907064b84a8752c25c6fd5890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f1b63c203848ba95feb28f30b6b891" is_pre_election: true
I20251025 08:02:46.641124 3347 leader_election.cc:304] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e8418b907064b84a8752c25c6fd5890; no voters: 384defb513fd42959725a27971ee3759, c5f1b63c203848ba95feb28f30b6b891
I20251025 08:02:46.641526 3548 raft_consensus.cc:2468] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 384defb513fd42959725a27971ee3759 in term 1.
I20251025 08:02:46.641526 3411 raft_consensus.cc:2468] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 384defb513fd42959725a27971ee3759 in term 1.
I20251025 08:02:46.641664 3769 raft_consensus.cc:2749] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 08:02:46.641860 3620 leader_election.cc:304] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384defb513fd42959725a27971ee3759, c5f1b63c203848ba95feb28f30b6b891; no voters:
I20251025 08:02:46.642040 3768 raft_consensus.cc:2804] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:46.642267 3768 raft_consensus.cc:697] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 1 LEADER]: Becoming Leader. State: Replica: 384defb513fd42959725a27971ee3759, State: Running, Role: LEADER
I20251025 08:02:46.642575 3768 consensus_queue.cc:237] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:46.645922 2467 catalog_manager.cc:5649] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 reported cstate change: term changed from 0 to 1, leader changed from <none> to 384defb513fd42959725a27971ee3759 (127.27.132.3). New cstate: current_term: 1 leader_uuid: "384defb513fd42959725a27971ee3759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } health_report { overall_health: UNKNOWN } } }
I20251025 08:02:46.676760 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.673535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.27.132.3:42201)
1025 08:02:46.673704 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.673711 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.673726 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:02:46.673807 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.673810 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.673863 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.673975 (+ 112us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:02:46.674459 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.674975 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.675671 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.675805 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.675895 (+ 90us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:02:46.676340 (+ 445us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:02:46.676457 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.676561 (+ 104us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.676609 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251025 08:02:46.682178 3548 raft_consensus.cc:1275] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891 [term 1 FOLLOWER]: Refusing update from remote peer 384defb513fd42959725a27971ee3759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:46.682219 3411 raft_consensus.cc:1275] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 1 FOLLOWER]: Refusing update from remote peer 384defb513fd42959725a27971ee3759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:46.682824 3768 consensus_queue.cc:1048] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:46.683226 3772 consensus_queue.cc:1048] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:46.691632 3777 mvcc.cc:204] Tried to move back new op lower bound from 7214609885923516416 to 7214609885770760192. Current Snapshot: MvccSnapshot[applied={T|T < 7214609885923516416}]
I20251025 08:02:46.701015 2469 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35042:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:46.701160 2469 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35042:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:46.705791 2469 catalog_manager.cc:5936] T 00000000000000000000000000000000 P d28b5159b5704014b011a9766e6f0e8c: Sending DeleteTablet for 3 replicas of tablet 67a14d9714eb4c158aaf7a9997799942
I20251025 08:02:46.706802 3526 tablet_service.cc:1552] Processing DeleteTablet for tablet 67a14d9714eb4c158aaf7a9997799942 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:46 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:56072
I20251025 08:02:46.707422 3784 tablet_replica.cc:333] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: stopping tablet replica
I20251025 08:02:46.707932 3392 tablet_service.cc:1552] Processing DeleteTablet for tablet 67a14d9714eb4c158aaf7a9997799942 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:46 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:36526
I20251025 08:02:46.708667 3784 raft_consensus.cc:2243] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:46.708911 3665 tablet_service.cc:1552] Processing DeleteTablet for tablet 67a14d9714eb4c158aaf7a9997799942 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:46 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:56892
I20251025 08:02:46.709019 3784 raft_consensus.cc:2272] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:46.709780 3786 tablet_replica.cc:333] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759: stopping tablet replica
I20251025 08:02:46.710045 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 2438
I20251025 08:02:46.710039 3786 raft_consensus.cc:2243] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:02:46.710620 3786 raft_consensus.cc:2272] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:46.710770 3784 ts_tablet_manager.cc:1916] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:46.711858 3786 ts_tablet_manager.cc:1916] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:46.713845 3784 ts_tablet_manager.cc:1929] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:46.714010 3784 log.cc:1199] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/67a14d9714eb4c158aaf7a9997799942
I20251025 08:02:46.714373 3786 ts_tablet_manager.cc:1929] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:46.714463 3786 log.cc:1199] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/67a14d9714eb4c158aaf7a9997799942
I20251025 08:02:46.714570 3784 ts_tablet_manager.cc:1950] T 67a14d9714eb4c158aaf7a9997799942 P c5f1b63c203848ba95feb28f30b6b891: Deleting consensus metadata
I20251025 08:02:46.714783 3786 ts_tablet_manager.cc:1950] T 67a14d9714eb4c158aaf7a9997799942 P 384defb513fd42959725a27971ee3759: Deleting consensus metadata
W20251025 08:02:46.718868 3618 connection.cc:537] server connection from 127.0.0.1:56892 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 08:02:46.718873 3483 connection.cc:537] server connection from 127.0.0.1:56072 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 08:02:46.719509 3785 tablet_replica.cc:333] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: stopping tablet replica
I20251025 08:02:46.719811 3785 raft_consensus.cc:2243] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:46.720199 3785 raft_consensus.cc:2272] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:46.721836 3785 ts_tablet_manager.cc:1916] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:46.724624 3785 ts_tablet_manager.cc:1929] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:46.724730 3785 log.cc:1199] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-0/wal/wals/67a14d9714eb4c158aaf7a9997799942
I20251025 08:02:46.725113 3785 ts_tablet_manager.cc:1950] T 67a14d9714eb4c158aaf7a9997799942 P 9e8418b907064b84a8752c25c6fd5890: Deleting consensus metadata
W20251025 08:02:46.726253 3348 connection.cc:441] server connection from 127.0.0.1:36526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36526 (request call id 1) could send its response
I20251025 08:02:46.731973 3747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.722214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.27.132.61:34389)
1025 08:02:46.722348 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.722355 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.722901 (+ 546us) server_negotiation.cc:408] Connection header received
1025 08:02:46.723017 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.723021 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.723082 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.723181 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.724307 (+ 1126us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.725114 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.726129 (+ 1015us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.726323 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.728129 (+ 1806us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.728148 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.728151 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.728181 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.730219 (+ 2038us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.730706 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.730712 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.730713 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.730758 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.731006 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.731010 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.731012 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.731196 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.731297 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.731698 (+ 401us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.731812 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251025 08:02:46.732270 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:46.722558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.27.132.60:37999)
1025 08:02:46.722681 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:46.722686 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:46.723572 (+ 886us) server_negotiation.cc:408] Connection header received
1025 08:02:46.723777 (+ 205us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:46.723780 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:46.723834 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:46.723887 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:46.725415 (+ 1528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.726123 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:46.727138 (+ 1015us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:46.727698 (+ 560us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:46.728782 (+ 1084us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:46.728801 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:46.728803 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:46.728827 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:46.730522 (+ 1695us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.730960 (+ 438us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.730966 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.730969 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.731018 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:46.731253 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:46.731256 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:46.731258 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:46.731462 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:46.731581 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:46.731999 (+ 418us) server_negotiation.cc:300] Negotiation successful
1025 08:02:46.732117 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 08:02:47.695814 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:47.695243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.2:34691)
1025 08:02:47.695574 (+ 331us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:47.695694 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":94,"threads_started":1}
W20251025 08:02:47.696180 3595 heartbeater.cc:646] Failed to heartbeat to 127.27.132.62:42629 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
I20251025 08:02:47.699110 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:47.698699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.1:45007)
1025 08:02:47.698945 (+ 246us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:47.699025 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
W20251025 08:02:47.699411 3459 heartbeater.cc:646] Failed to heartbeat to 127.27.132.62:42629 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
I20251025 08:02:47.699416 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:47.698962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.3:46149)
1025 08:02:47.699203 (+ 241us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:47.699280 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
W20251025 08:02:47.699652 3732 heartbeater.cc:646] Failed to heartbeat to 127.27.132.62:42629 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
I20251025 08:02:48.206594 3797 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d28b5159b5704014b011a9766e6f0e8c)
I20251025 08:02:48.206789 3797 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } }
I20251025 08:02:48.207142 3798 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d28b5159b5704014b011a9766e6f0e8c)
I20251025 08:02:48.207316 3798 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } }
I20251025 08:02:48.207966 3797 leader_election.cc:290] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629), 7fb81dc334b641e084097ce2feabe4a8 (127.27.132.61:34389)
I20251025 08:02:48.208323 3798 leader_election.cc:290] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629), 26d7cf36796744208fc9d9bf1d960347 (127.27.132.60:37999)
I20251025 08:02:48.208449 3092 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26d7cf36796744208fc9d9bf1d960347" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7fb81dc334b641e084097ce2feabe4a8" is_pre_election: true
I20251025 08:02:48.208602 3092 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d7cf36796744208fc9d9bf1d960347 in term 1.
I20251025 08:02:48.208699 3271 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7fb81dc334b641e084097ce2feabe4a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "26d7cf36796744208fc9d9bf1d960347" is_pre_election: true
I20251025 08:02:48.208741 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:48.208097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:44648)
1025 08:02:48.208541 (+ 444us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:48.208621 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":60,"threads_started":1}
I20251025 08:02:48.208874 3271 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb81dc334b641e084097ce2feabe4a8 in term 1.
W20251025 08:02:48.208922 3251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
I20251025 08:02:48.209013 3250 leader_election.cc:304] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d7cf36796744208fc9d9bf1d960347, 7fb81dc334b641e084097ce2feabe4a8; no voters:
I20251025 08:02:48.209236 3797 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 08:02:48.209232 3066 leader_election.cc:304] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d7cf36796744208fc9d9bf1d960347, 7fb81dc334b641e084097ce2feabe4a8; no voters:
I20251025 08:02:48.209295 3797 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d28b5159b5704014b011a9766e6f0e8c)
I20251025 08:02:48.209343 3797 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 1 FOLLOWER]: Advancing to term 2
I20251025 08:02:48.209266 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:48.208428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:44660)
1025 08:02:48.208988 (+ 560us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:48.209166 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":58,"threads_started":1}
I20251025 08:02:48.209453 3798 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 08:02:48.209514 3798 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d28b5159b5704014b011a9766e6f0e8c)
I20251025 08:02:48.209574 3798 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 1 FOLLOWER]: Advancing to term 2
W20251025 08:02:48.209596 3067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
I20251025 08:02:48.209903 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:48.209633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:44670)
1025 08:02:48.209786 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:48.209833 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20251025 08:02:48.210029 3251 leader_election.cc:336] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629): Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
I20251025 08:02:48.210613 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:48.210381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:44682)
1025 08:02:48.210518 (+ 137us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:48.210565 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20251025 08:02:48.210598 3797 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } }
I20251025 08:02:48.210598 3798 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } }
W20251025 08:02:48.210820 3067 leader_election.cc:336] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629): Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
I20251025 08:02:48.210964 3798 leader_election.cc:290] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [CANDIDATE]: Term 2 election: Requested vote from peers d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629), 26d7cf36796744208fc9d9bf1d960347 (127.27.132.60:37999)
I20251025 08:02:48.211314 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:48.211038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:44690)
1025 08:02:48.211187 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:48.211236 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20251025 08:02:48.211552 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:48.211156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:44700)
1025 08:02:48.211432 (+ 276us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:48.211486 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20251025 08:02:48.211313 3271 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7fb81dc334b641e084097ce2feabe4a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "26d7cf36796744208fc9d9bf1d960347"
I20251025 08:02:48.211679 3271 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb81dc334b641e084097ce2feabe4a8 in current term 2: Already voted for candidate 26d7cf36796744208fc9d9bf1d960347 in this term.
I20251025 08:02:48.211711 3092 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26d7cf36796744208fc9d9bf1d960347" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7fb81dc334b641e084097ce2feabe4a8"
I20251025 08:02:48.211853 3092 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 26d7cf36796744208fc9d9bf1d960347 in current term 2: Already voted for candidate 7fb81dc334b641e084097ce2feabe4a8 in this term.
I20251025 08:02:48.212146 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:48.211904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:44704)
1025 08:02:48.212040 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:48.212087 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20251025 08:02:48.212224 3797 leader_election.cc:290] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 2 election: Requested vote from peers d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629), 7fb81dc334b641e084097ce2feabe4a8 (127.27.132.61:34389)
I20251025 08:02:48.212275 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:48.211973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:44718)
1025 08:02:48.212157 (+ 184us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:48.212209 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20251025 08:02:48.212275 3251 leader_election.cc:336] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629): Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
I20251025 08:02:48.212330 3251 leader_election.cc:304] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26d7cf36796744208fc9d9bf1d960347; no voters: 7fb81dc334b641e084097ce2feabe4a8, d28b5159b5704014b011a9766e6f0e8c
W20251025 08:02:48.212352 3067 leader_election.cc:336] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629): Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
I20251025 08:02:48.212388 3067 leader_election.cc:304] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb81dc334b641e084097ce2feabe4a8; no voters: 26d7cf36796744208fc9d9bf1d960347, d28b5159b5704014b011a9766e6f0e8c
I20251025 08:02:48.212527 3798 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 08:02:48.212548 3797 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251025 08:02:48.697644 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:48.696885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.2:43847)
1025 08:02:48.697200 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:48.697275 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
I20251025 08:02:48.700320 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:48.699871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.1:44395)
1025 08:02:48.700143 (+ 272us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:48.700209 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20251025 08:02:48.700516 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:48.700137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.3:53175)
1025 08:02:48.700355 (+ 218us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:48.700416 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20251025 08:02:49.699608 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:49.698845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.2:32929)
1025 08:02:49.699416 (+ 571us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:49.699490 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":92,"threads_started":1}
I20251025 08:02:49.701342 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:49.700914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.1:54899)
1025 08:02:49.701205 (+ 291us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:49.701263 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":74,"threads_started":1}
I20251025 08:02:49.701428 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:49.701069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.3:38647)
1025 08:02:49.701294 (+ 225us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:49.701344 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20251025 08:02:50.307876 3809 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:02:50.308017 3809 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } }
I20251025 08:02:50.308475 3809 leader_election.cc:290] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629), 7fb81dc334b641e084097ce2feabe4a8 (127.27.132.61:34389)
I20251025 08:02:50.308979 3092 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26d7cf36796744208fc9d9bf1d960347" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7fb81dc334b641e084097ce2feabe4a8" is_pre_election: true
I20251025 08:02:50.309111 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.308683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:45024)
1025 08:02:50.308925 (+ 242us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.308994 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
I20251025 08:02:50.309144 3092 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d7cf36796744208fc9d9bf1d960347 in term 2.
I20251025 08:02:50.309511 3250 leader_election.cc:304] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d7cf36796744208fc9d9bf1d960347, 7fb81dc334b641e084097ce2feabe4a8; no voters:
I20251025 08:02:50.309700 3809 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251025 08:02:50.309759 3809 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:02:50.309808 3809 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 2 FOLLOWER]: Advancing to term 3
I20251025 08:02:50.310081 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.309818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:45030)
1025 08:02:50.309952 (+ 134us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.310010 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20251025 08:02:50.310210 3251 leader_election.cc:336] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629): Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
I20251025 08:02:50.310923 3809 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } }
I20251025 08:02:50.311252 3809 leader_election.cc:290] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 3 election: Requested vote from peers d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629), 7fb81dc334b641e084097ce2feabe4a8 (127.27.132.61:34389)
I20251025 08:02:50.311625 3092 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26d7cf36796744208fc9d9bf1d960347" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7fb81dc334b641e084097ce2feabe4a8"
I20251025 08:02:50.311728 3092 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 2 FOLLOWER]: Advancing to term 3
I20251025 08:02:50.311759 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.311483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:45042)
1025 08:02:50.311627 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.311683 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20251025 08:02:50.312343 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.312154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:45046)
1025 08:02:50.312252 (+ 98us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.312298 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251025 08:02:50.312482 3251 leader_election.cc:336] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629): Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
I20251025 08:02:50.312671 3092 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d7cf36796744208fc9d9bf1d960347 in term 3.
I20251025 08:02:50.312959 3250 leader_election.cc:304] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26d7cf36796744208fc9d9bf1d960347, 7fb81dc334b641e084097ce2feabe4a8; no voters: d28b5159b5704014b011a9766e6f0e8c
I20251025 08:02:50.313146 3809 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 3 FOLLOWER]: Leader election won for term 3
I20251025 08:02:50.313311 3809 raft_consensus.cc:697] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [term 3 LEADER]: Becoming Leader. State: Replica: 26d7cf36796744208fc9d9bf1d960347, State: Running, Role: LEADER
I20251025 08:02:50.313556 3809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } }
I20251025 08:02:50.314814 3812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d7cf36796744208fc9d9bf1d960347. Latest consensus state: current_term: 3 leader_uuid: "26d7cf36796744208fc9d9bf1d960347" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:50.314899 3812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:50.315150 3814 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:50.317267 3814 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:50.317673 3814 catalog_manager.cc:1269] Loaded cluster ID: 73f659919e404a6591f3f9e2aa29e7c7
I20251025 08:02:50.317720 3814 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:50.318241 3814 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:50.318626 3814 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: Loaded TSK: 0
I20251025 08:02:50.319351 3814 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 08:02:50.363503 3265 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:02:50.365947 3265 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:02:50.368948 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.368668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:45062)
1025 08:02:50.368817 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.368872 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20251025 08:02:50.369000 3092 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [term 3 FOLLOWER]: Refusing update from remote peer 26d7cf36796744208fc9d9bf1d960347: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20251025 08:02:50.369467 3809 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20251025 08:02:50.369684 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.369447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.0.0.1:45066)
1025 08:02:50.369561 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.369619 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20251025 08:02:50.369786 3251 consensus_peers.cc:597] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 -> Peer d28b5159b5704014b011a9766e6f0e8c (127.27.132.62:42629): Couldn't send request to peer d28b5159b5704014b011a9766e6f0e8c. Status: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 08:02:50.370872 3816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d7cf36796744208fc9d9bf1d960347. Latest consensus state: current_term: 3 leader_uuid: "26d7cf36796744208fc9d9bf1d960347" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:50.370995 3816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 08:02:50.372047 3809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "26d7cf36796744208fc9d9bf1d960347" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:50.372154 3809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347 [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:50.372606 3816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "26d7cf36796744208fc9d9bf1d960347" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d28b5159b5704014b011a9766e6f0e8c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42629 } } peers { permanent_uuid: "7fb81dc334b641e084097ce2feabe4a8" member_type: VOTER last_known_addr { host: "127.27.132.61" port: 34389 } attrs { promote: false } } peers { permanent_uuid: "26d7cf36796744208fc9d9bf1d960347" member_type: VOTER last_known_addr { host: "127.27.132.60" port: 37999 } attrs { promote: false } } }
I20251025 08:02:50.372753 3816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb81dc334b641e084097ce2feabe4a8 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 08:02:50.381711 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.378663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:41457 (local address 127.0.0.1:57234)
1025 08:02:50.379049 (+ 386us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.379062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:50.379140 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:50.379411 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:50.379414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:50.379433 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:50.379675 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:50.379685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.380656 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:50.380661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:50.381320 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:50.381329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.381443 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:50.381465 (+ 22us) client_negotiation.cc:769] Sending connection context
1025 08:02:50.381519 (+ 54us) client_negotiation.cc:240] Negotiation successful
1025 08:02:50.381576 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":98,"threads_started":1}
I20251025 08:02:50.381995 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.377965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:42201 (local address 127.0.0.1:48782)
1025 08:02:50.378156 (+ 191us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.378172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:50.378260 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:50.379293 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:50.379298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:50.379318 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:50.379603 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:50.379611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.380647 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:50.380651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:50.381638 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:50.381648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.381755 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:50.381771 (+ 16us) client_negotiation.cc:769] Sending connection context
1025 08:02:50.381821 (+ 50us) client_negotiation.cc:240] Negotiation successful
1025 08:02:50.381877 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20251025 08:02:50.382278 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.378783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.27.132.2:41457)
1025 08:02:50.379145 (+ 362us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:50.379150 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:50.379165 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:02:50.379217 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:50.379220 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:50.379275 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:50.379398 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:50.379782 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.380545 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:50.381457 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.381959 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:50.381998 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:50.382052 (+ 54us) server_negotiation.cc:300] Negotiation successful
1025 08:02:50.382132 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":81,"threads_started":1}
I20251025 08:02:50.382779 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.378995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.27.132.1:46805)
1025 08:02:50.379506 (+ 511us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:50.379511 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:50.379526 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:02:50.379576 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:50.379580 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:50.379624 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:50.379712 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:50.380117 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.380606 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:50.381957 (+ 1351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.382488 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:50.382528 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:50.382581 (+ 53us) server_negotiation.cc:300] Negotiation successful
1025 08:02:50.382646 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":71,"threads_started":1}
I20251025 08:02:50.382889 3526 tablet_service.cc:1505] Processing CreateTablet for tablet a2a80de51ae741dabc8142853411bbaf (DEFAULT_TABLE table=test-table [id=fde6441fb89b443185cfd04ac98f432b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:50.383033 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.378314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:46805 (local address 127.0.0.1:37632)
1025 08:02:50.378725 (+ 411us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.378742 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:50.378827 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:50.379832 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:50.379835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:50.379849 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:50.380001 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:50.380005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.380707 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:50.380710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:50.381638 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:50.381648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.381755 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:50.381771 (+ 16us) client_negotiation.cc:769] Sending connection context
1025 08:02:50.381821 (+ 50us) client_negotiation.cc:240] Negotiation successful
1025 08:02:50.381873 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":116,"threads_started":1}
I20251025 08:02:50.383181 3526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a80de51ae741dabc8142853411bbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:50.383615 3392 tablet_service.cc:1505] Processing CreateTablet for tablet a2a80de51ae741dabc8142853411bbaf (DEFAULT_TABLE table=test-table [id=fde6441fb89b443185cfd04ac98f432b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:50.383848 3821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.378100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48782 (local address 127.27.132.3:42201)
1025 08:02:50.379077 (+ 977us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:50.379084 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:50.379100 (+ 16us) server_negotiation.cc:408] Connection header received
1025 08:02:50.379154 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:50.379158 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:50.379199 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:50.379274 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:50.379721 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.380508 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:50.383022 (+ 2514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.383560 (+ 538us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:50.383595 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:50.383646 (+ 51us) server_negotiation.cc:300] Negotiation successful
1025 08:02:50.383720 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":70,"threads_started":1}
I20251025 08:02:50.383884 3392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a80de51ae741dabc8142853411bbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:50.386101 3665 tablet_service.cc:1505] Processing CreateTablet for tablet a2a80de51ae741dabc8142853411bbaf (DEFAULT_TABLE table=test-table [id=fde6441fb89b443185cfd04ac98f432b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:50.386356 3827 tablet_bootstrap.cc:492] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890: Bootstrap starting.
I20251025 08:02:50.386372 3665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a80de51ae741dabc8142853411bbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:50.386520 3826 tablet_bootstrap.cc:492] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891: Bootstrap starting.
I20251025 08:02:50.387140 3827 tablet_bootstrap.cc:654] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:50.387477 3826 tablet_bootstrap.cc:654] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:50.388305 3827 tablet_bootstrap.cc:492] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890: No bootstrap required, opened a new log
I20251025 08:02:50.388411 3827 ts_tablet_manager.cc:1403] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:50.388681 3826 tablet_bootstrap.cc:492] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891: No bootstrap required, opened a new log
I20251025 08:02:50.388687 3830 tablet_bootstrap.cc:492] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759: Bootstrap starting.
I20251025 08:02:50.388868 3826 ts_tablet_manager.cc:1403] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:50.388968 3827 raft_consensus.cc:359] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:50.389084 3827 raft_consensus.cc:385] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:50.389158 3827 raft_consensus.cc:740] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8418b907064b84a8752c25c6fd5890, State: Initialized, Role: FOLLOWER
I20251025 08:02:50.389299 3827 consensus_queue.cc:260] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:50.389302 3826 raft_consensus.cc:359] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:50.389397 3826 raft_consensus.cc:385] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:50.389454 3826 raft_consensus.cc:740] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f1b63c203848ba95feb28f30b6b891, State: Initialized, Role: FOLLOWER
I20251025 08:02:50.389658 3826 consensus_queue.cc:260] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:50.389880 3830 tablet_bootstrap.cc:654] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:50.390071 3826 ts_tablet_manager.cc:1434] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:02:50.391218 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.390811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.2:49099)
1025 08:02:50.391070 (+ 259us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.391126 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20251025 08:02:50.391698 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.390065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.1:52709)
1025 08:02:50.391541 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.391596 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1390,"thread_start_us":92,"threads_started":1}
I20251025 08:02:50.392530 3827 ts_tablet_manager.cc:1434] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 08:02:50.392724 3458 heartbeater.cc:499] Master 127.27.132.60:37999 was elected leader, sending a full tablet report...
I20251025 08:02:50.392779 3594 heartbeater.cc:499] Master 127.27.132.60:37999 was elected leader, sending a full tablet report...
I20251025 08:02:50.393240 3830 tablet_bootstrap.cc:492] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759: No bootstrap required, opened a new log
I20251025 08:02:50.393340 3830 ts_tablet_manager.cc:1403] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20251025 08:02:50.393761 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.392936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.1:41183)
1025 08:02:50.393095 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.393148 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20251025 08:02:50.393863 3830 raft_consensus.cc:359] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:50.394183 3830 raft_consensus.cc:385] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:50.394236 3830 raft_consensus.cc:740] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384defb513fd42959725a27971ee3759, State: Initialized, Role: FOLLOWER
I20251025 08:02:50.394379 3830 consensus_queue.cc:260] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:50.394929 3830 ts_tablet_manager.cc:1434] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:50.395641 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.393162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.2:52625)
1025 08:02:50.395523 (+ 2361us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.395575 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2281}
I20251025 08:02:50.395519 3836 raft_consensus.cc:493] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:02:50.396281 3836 raft_consensus.cc:515] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:50.396351 3731 heartbeater.cc:499] Master 127.27.132.60:37999 was elected leader, sending a full tablet report...
I20251025 08:02:50.395985 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.395265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.3:55805)
1025 08:02:50.395682 (+ 417us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.395884 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":139,"threads_started":1}
I20251025 08:02:50.396631 3836 leader_election.cc:290] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e8418b907064b84a8752c25c6fd5890 (127.27.132.1:46805), c5f1b63c203848ba95feb28f30b6b891 (127.27.132.2:41457)
I20251025 08:02:50.396975 3411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2a80de51ae741dabc8142853411bbaf" candidate_uuid: "384defb513fd42959725a27971ee3759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8418b907064b84a8752c25c6fd5890" is_pre_election: true
I20251025 08:02:50.397107 3411 raft_consensus.cc:2468] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 384defb513fd42959725a27971ee3759 in term 0.
I20251025 08:02:50.397303 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.396912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.3:46895)
1025 08:02:50.397178 (+ 266us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.397226 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
I20251025 08:02:50.397444 3620 leader_election.cc:304] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384defb513fd42959725a27971ee3759, 9e8418b907064b84a8752c25c6fd5890; no voters:
I20251025 08:02:50.397542 3548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2a80de51ae741dabc8142853411bbaf" candidate_uuid: "384defb513fd42959725a27971ee3759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f1b63c203848ba95feb28f30b6b891" is_pre_election: true
I20251025 08:02:50.397622 3836 raft_consensus.cc:2804] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:02:50.397643 3548 raft_consensus.cc:2468] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 384defb513fd42959725a27971ee3759 in term 0.
I20251025 08:02:50.397677 3836 raft_consensus.cc:493] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:02:50.397706 3836 raft_consensus.cc:3060] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:50.398571 3836 raft_consensus.cc:515] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:50.398857 3836 leader_election.cc:290] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [CANDIDATE]: Term 1 election: Requested vote from peers 9e8418b907064b84a8752c25c6fd5890 (127.27.132.1:46805), c5f1b63c203848ba95feb28f30b6b891 (127.27.132.2:41457)
I20251025 08:02:50.399144 3411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2a80de51ae741dabc8142853411bbaf" candidate_uuid: "384defb513fd42959725a27971ee3759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8418b907064b84a8752c25c6fd5890"
I20251025 08:02:50.399201 3548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2a80de51ae741dabc8142853411bbaf" candidate_uuid: "384defb513fd42959725a27971ee3759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f1b63c203848ba95feb28f30b6b891"
I20251025 08:02:50.399286 3411 raft_consensus.cc:3060] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:50.399289 3548 raft_consensus.cc:3060] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:50.400298 3411 raft_consensus.cc:2468] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 384defb513fd42959725a27971ee3759 in term 1.
I20251025 08:02:50.400298 3548 raft_consensus.cc:2468] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 384defb513fd42959725a27971ee3759 in term 1.
I20251025 08:02:50.400602 3620 leader_election.cc:304] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384defb513fd42959725a27971ee3759, 9e8418b907064b84a8752c25c6fd5890; no voters:
I20251025 08:02:50.400790 3836 raft_consensus.cc:2804] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:50.400978 3836 raft_consensus.cc:697] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 1 LEADER]: Becoming Leader. State: Replica: 384defb513fd42959725a27971ee3759, State: Running, Role: LEADER
I20251025 08:02:50.401127 3836 consensus_queue.cc:237] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } }
I20251025 08:02:50.401654 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.401396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.3:34897)
1025 08:02:50.401535 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.401580 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251025 08:02:50.402696 3266 catalog_manager.cc:5649] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 reported cstate change: term changed from 0 to 1, leader changed from <none> to 384defb513fd42959725a27971ee3759 (127.27.132.3). New cstate: current_term: 1 leader_uuid: "384defb513fd42959725a27971ee3759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384defb513fd42959725a27971ee3759" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 } health_report { overall_health: UNKNOWN } } }
I20251025 08:02:50.414479 3821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.411769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.27.132.3:42201)
1025 08:02:50.411933 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:50.411938 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:50.411952 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:02:50.412033 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:50.412038 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:50.412088 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:50.412154 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:02:50.412492 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.413076 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:50.413792 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:50.413985 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:50.414029 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:02:50.414097 (+ 68us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:02:50.414175 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:50.414285 (+ 110us) server_negotiation.cc:300] Negotiation successful
1025 08:02:50.414331 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 08:02:50.416611 3411 raft_consensus.cc:1275] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890 [term 1 FOLLOWER]: Refusing update from remote peer 384defb513fd42959725a27971ee3759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:50.416695 3548 raft_consensus.cc:1275] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891 [term 1 FOLLOWER]: Refusing update from remote peer 384defb513fd42959725a27971ee3759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:50.417076 3836 consensus_queue.cc:1048] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8418b907064b84a8752c25c6fd5890" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:50.417204 3839 consensus_queue.cc:1048] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f1b63c203848ba95feb28f30b6b891" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 41457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:50.419600 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.419222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.1:39823)
1025 08:02:50.419455 (+ 233us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.419521 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20251025 08:02:50.419770 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.419503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.2:37295)
1025 08:02:50.419627 (+ 124us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.419679 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251025 08:02:50.421424 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.421150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.1:54319)
1025 08:02:50.421309 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.421358 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20251025 08:02:50.422461 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.422184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.3:45023)
1025 08:02:50.422329 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.422387 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20251025 08:02:50.423041 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.422787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42629 (local address 127.27.132.2:54495)
1025 08:02:50.422925 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.422977 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20251025 08:02:50.422938 3843 mvcc.cc:204] Tried to move back new op lower bound from 7214609901221969920 to 7214609901164085248. Current Snapshot: MvccSnapshot[applied={T|T < 7214609901221969920}]
I20251025 08:02:50.428882 3266 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37896:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:50.429040 3266 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37896:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:50.432495 3266 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 26d7cf36796744208fc9d9bf1d960347: Sending DeleteTablet for 3 replicas of tablet a2a80de51ae741dabc8142853411bbaf
I20251025 08:02:50.433066 3665 tablet_service.cc:1552] Processing DeleteTablet for tablet a2a80de51ae741dabc8142853411bbaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:50 UTC) from {username='oryx', principal='oryx/127.27.132.60@KRBTEST.COM'} at 127.0.0.1:48782
I20251025 08:02:50.433384 3852 tablet_replica.cc:333] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759: stopping tablet replica
I20251025 08:02:50.433427 3392 tablet_service.cc:1552] Processing DeleteTablet for tablet a2a80de51ae741dabc8142853411bbaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:50 UTC) from {username='oryx', principal='oryx/127.27.132.60@KRBTEST.COM'} at 127.0.0.1:37632
I20251025 08:02:50.433418 3526 tablet_service.cc:1552] Processing DeleteTablet for tablet a2a80de51ae741dabc8142853411bbaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:50 UTC) from {username='oryx', principal='oryx/127.27.132.60@KRBTEST.COM'} at 127.0.0.1:57234
I20251025 08:02:50.433588 3852 raft_consensus.cc:2243] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:02:50.433765 3853 tablet_replica.cc:333] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890: stopping tablet replica
I20251025 08:02:50.433796 3852 raft_consensus.cc:2272] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:50.434011 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 3329
I20251025 08:02:50.433907 3853 raft_consensus.cc:2243] T a2a80de51ae741dabc8142853411bbaf P 9e8418b907064b84a8752c25c6fd5890 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:50.434106 3854 tablet_replica.cc:333] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891: stopping tablet replica
I20251025 08:02:50.434196 3854 raft_consensus.cc:2243] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:50.434295 3854 raft_consensus.cc:2272] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:50.434971 3852 ts_tablet_manager.cc:1916] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:50.435076 3854 ts_tablet_manager.cc:1916] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:50.436853 3854 ts_tablet_manager.cc:1929] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:50.436914 3854 log.cc:1199] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/a2a80de51ae741dabc8142853411bbaf
I20251025 08:02:50.437150 3854 ts_tablet_manager.cc:1950] T a2a80de51ae741dabc8142853411bbaf P c5f1b63c203848ba95feb28f30b6b891: Deleting consensus metadata
I20251025 08:02:50.437736 3252 catalog_manager.cc:4985] TS c5f1b63c203848ba95feb28f30b6b891 (127.27.132.2:41457): tablet a2a80de51ae741dabc8142853411bbaf (table test-table [id=fde6441fb89b443185cfd04ac98f432b]) successfully deleted
I20251025 08:02:50.438004 3852 ts_tablet_manager.cc:1929] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:50.438076 3852 log.cc:1199] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/a2a80de51ae741dabc8142853411bbaf
I20251025 08:02:50.438359 3852 ts_tablet_manager.cc:1950] T a2a80de51ae741dabc8142853411bbaf P 384defb513fd42959725a27971ee3759: Deleting consensus metadata
I20251025 08:02:50.439056 3250 catalog_manager.cc:4985] TS 384defb513fd42959725a27971ee3759 (127.27.132.3:42201): tablet a2a80de51ae741dabc8142853411bbaf (table test-table [id=fde6441fb89b443185cfd04ac98f432b]) successfully deleted
I20251025 08:02:50.443838 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 3465
I20251025 08:02:50.444669 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:50.443959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:46805 (local address 127.0.0.1:37642)
1025 08:02:50.444499 (+ 540us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:50.444565 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.1:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408}
W20251025 08:02:50.444846 3252 catalog_manager.cc:4712] TS 9e8418b907064b84a8752c25c6fd5890 (127.27.132.1:46805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2a80de51ae741dabc8142853411bbaf: Network error: Client connection negotiation failed: client connection to 127.27.132.1:46805: connect: Connection refused (error 111)
I20251025 08:02:50.450049 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 3602
I20251025 08:02:50.456874 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 2943
I20251025 08:02:50.462815 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 3046
2025-10-25T08:02:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9349 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:02:50 dist-test-slave-lq8l krb5kdc[3857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:02:50 dist-test-slave-lq8l krb5kdc[3857](info): set up 2 sockets
Oct 25 08:02:50 dist-test-slave-lq8l krb5kdc[3857](info): commencing operation
krb5kdc: starting...
W20251025 08:02:52.444780 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.958s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:02:52 dist-test-slave-lq8l krb5kdc[3857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:02:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
I20251025 08:02:52.583803 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:34111
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:52.689589 3873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:52.689832 3873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:52.689921 3873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:52.693113 3873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:52.693178 3873 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251025 08:02:52.693212 3873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:52.693248 3873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:52.693290 3873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:52.693329 3873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:52.697504 3873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:34111
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:34111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:52.698612 3873 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:52.699525 3873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251025 08:02:52.705410 3873 server_base.cc:1047] running on GCE node
W20251025 08:02:52.705425 3881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:52.705324 3878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:52.705324 3879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:52.706063 3873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:52.706954 3873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:52.708105 3873 hybrid_clock.cc:648] HybridClock initialized: now 1761379372708088 us; error 34 us; skew 500 ppm
Oct 25 08:02:52 dist-test-slave-lq8l krb5kdc[3857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379372, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:52.710668 3873 init.cc:377] Logged in from keytab as oryx/127.27.132.62@KRBTEST.COM (short username oryx)
I20251025 08:02:52.711779 3873 webserver.cc:492] Webserver started at http://127.27.132.62:35737/ using document root <none> and password file <none>
I20251025 08:02:52.712294 3873 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:52.712339 3873 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:52.712493 3873 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:52.714123 3873 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "46c0e57e26994ed8909a1c92d2a7c5a3"
format_stamp: "Formatted at 2025-10-25 08:02:52 on dist-test-slave-lq8l"
server_key: "612a1b989a809fc30566551a55859147"
server_key_iv: "bcaba71d10321344f65dce4019a9a96e"
server_key_version: "encryptionkey@0"
I20251025 08:02:52.714545 3873 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "46c0e57e26994ed8909a1c92d2a7c5a3"
format_stamp: "Formatted at 2025-10-25 08:02:52 on dist-test-slave-lq8l"
server_key: "612a1b989a809fc30566551a55859147"
server_key_iv: "bcaba71d10321344f65dce4019a9a96e"
server_key_version: "encryptionkey@0"
I20251025 08:02:52.717881 3873 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251025 08:02:52.720201 3888 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:52.721252 3873 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 08:02:52.721356 3873 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "46c0e57e26994ed8909a1c92d2a7c5a3"
format_stamp: "Formatted at 2025-10-25 08:02:52 on dist-test-slave-lq8l"
server_key: "612a1b989a809fc30566551a55859147"
server_key_iv: "bcaba71d10321344f65dce4019a9a96e"
server_key_version: "encryptionkey@0"
I20251025 08:02:52.721472 3873 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:52.733647 3873 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:52.740999 3873 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:52.741185 3873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:52.749716 3873 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:34111
I20251025 08:02:52.749933 3950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:34111 every 8 connection(s)
I20251025 08:02:52.750826 3873 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:52.753618 3951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:52.758637 3951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3: Bootstrap starting.
I20251025 08:02:52.759454 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 3873
I20251025 08:02:52.759565 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:52.759804 28176 external_mini_cluster.cc:1468] Setting key 4b0031b2b0aab5e92f4c7f307fafbb6d
I20251025 08:02:52.761265 3951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:52.761973 3951 log.cc:826] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:52.764024 3951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3: No bootstrap required, opened a new log
Oct 25 08:02:52 dist-test-slave-lq8l krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:52.767393 3951 raft_consensus.cc:359] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c0e57e26994ed8909a1c92d2a7c5a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34111 } }
I20251025 08:02:52.767592 3951 raft_consensus.cc:385] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:52.767656 3951 raft_consensus.cc:740] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c0e57e26994ed8909a1c92d2a7c5a3, State: Initialized, Role: FOLLOWER
I20251025 08:02:52.768085 3951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c0e57e26994ed8909a1c92d2a7c5a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34111 } }
I20251025 08:02:52.768210 3951 raft_consensus.cc:399] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:52.768275 3951 raft_consensus.cc:493] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:52.768368 3951 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:52.769182 3951 raft_consensus.cc:515] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c0e57e26994ed8909a1c92d2a7c5a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34111 } }
I20251025 08:02:52.769572 3951 leader_election.cc:304] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46c0e57e26994ed8909a1c92d2a7c5a3; no voters:
I20251025 08:02:52.769922 3951 leader_election.cc:290] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:52.770025 3956 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:52.770238 3956 raft_consensus.cc:697] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [term 1 LEADER]: Becoming Leader. State: Replica: 46c0e57e26994ed8909a1c92d2a7c5a3, State: Running, Role: LEADER
I20251025 08:02:52.770591 3956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c0e57e26994ed8909a1c92d2a7c5a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34111 } }
I20251025 08:02:52.771140 3951 sys_catalog.cc:565] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:52.771821 3956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46c0e57e26994ed8909a1c92d2a7c5a3. Latest consensus state: current_term: 1 leader_uuid: "46c0e57e26994ed8909a1c92d2a7c5a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c0e57e26994ed8909a1c92d2a7c5a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34111 } } }
I20251025 08:02:52.772529 3956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:52.772423 3957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46c0e57e26994ed8909a1c92d2a7c5a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c0e57e26994ed8909a1c92d2a7c5a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34111 } } }
I20251025 08:02:52.772850 3957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3 [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:52.774144 3965 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:52.774308 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:52.760769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.27.132.62:34111)
1025 08:02:52.761214 (+ 445us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:52.761225 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:52.761259 (+ 34us) server_negotiation.cc:408] Connection header received
1025 08:02:52.761820 (+ 561us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:52.761838 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:52.762158 (+ 320us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:52.762510 (+ 352us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:52.763452 (+ 942us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:52.764229 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:52.765074 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:52.765291 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:52.767766 (+ 2475us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:52.767792 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:52.767804 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:52.767838 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:52.770546 (+ 2708us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:52.771045 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:52.771050 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:52.771056 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:52.771153 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:52.771524 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:52.771529 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:52.771533 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:52.771998 (+ 465us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:52.772954 (+ 956us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:52.773149 (+ 195us) server_negotiation.cc:300] Negotiation successful
1025 08:02:52.773408 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":153,"threads_started":1}
I20251025 08:02:52.777181 3965 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:52.782756 3965 catalog_manager.cc:1357] Generated new cluster ID: aa1126d495214f58a7f0a57452de56f1
I20251025 08:02:52.782831 3965 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:52.788874 3965 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:52.790117 3965 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:52.803099 3965 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3: Generated new TSK 0
I20251025 08:02:52.803959 3965 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
I20251025 08:02:52.856531 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34111
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:52.964226 3979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:52.964444 3979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:52.964505 3979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:52.967556 3979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:52.967619 3979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:52.967672 3979 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 08:02:52.967713 3979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:02:52.971899 3979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.132.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:52.972998 3979 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:52.973889 3979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251025 08:02:52.980420 3979 server_base.cc:1047] running on GCE node
W20251025 08:02:52.980371 3987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:52.980368 3984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:52.980587 3985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:52.980922 3979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:52.981499 3979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:52.982671 3979 hybrid_clock.cc:648] HybridClock initialized: now 1761379372982655 us; error 30 us; skew 500 ppm
Oct 25 08:02:52 dist-test-slave-lq8l krb5kdc[3857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379372, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:52.985360 3979 init.cc:377] Logged in from keytab as oryx/127.27.132.1@KRBTEST.COM (short username oryx)
I20251025 08:02:52.986400 3979 webserver.cc:492] Webserver started at http://127.27.132.1:37775/ using document root <none> and password file <none>
I20251025 08:02:52.986986 3979 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:52.987036 3979 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:52.987227 3979 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:52.988998 3979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "2826ef6b198d4fe79b769f1450049ec5"
format_stamp: "Formatted at 2025-10-25 08:02:52 on dist-test-slave-lq8l"
server_key: "a806b015f23233b8f37dc59967e0e2b9"
server_key_iv: "f05b3965dfad3825d53a31a6ebd7a3d1"
server_key_version: "encryptionkey@0"
I20251025 08:02:52.989497 3979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "2826ef6b198d4fe79b769f1450049ec5"
format_stamp: "Formatted at 2025-10-25 08:02:52 on dist-test-slave-lq8l"
server_key: "a806b015f23233b8f37dc59967e0e2b9"
server_key_iv: "f05b3965dfad3825d53a31a6ebd7a3d1"
server_key_version: "encryptionkey@0"
I20251025 08:02:52.992895 3979 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:02:52.994997 3994 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:52.996030 3979 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:02:52.996129 3979 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "2826ef6b198d4fe79b769f1450049ec5"
format_stamp: "Formatted at 2025-10-25 08:02:52 on dist-test-slave-lq8l"
server_key: "a806b015f23233b8f37dc59967e0e2b9"
server_key_iv: "f05b3965dfad3825d53a31a6ebd7a3d1"
server_key_version: "encryptionkey@0"
I20251025 08:02:52.996223 3979 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:53.024981 3979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:53.028127 3979 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:53.028290 3979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:53.029744 3979 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:53.029793 3979 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:53.029840 3979 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:53.029876 3979 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 08:02:53 dist-test-slave-lq8l krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379372, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:53.041126 3979 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:43221
I20251025 08:02:53.041265 4109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:43221 every 8 connection(s)
I20251025 08:02:53.042153 3979 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:02:53.042395 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.030449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:32971 (local address 127.27.132.62:34111)
1025 08:02:53.030575 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.030579 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.031426 (+ 847us) server_negotiation.cc:408] Connection header received
1025 08:02:53.032321 (+ 895us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.032326 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.032385 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.032492 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:53.034151 (+ 1659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.035034 (+ 883us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.035943 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.036135 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.038519 (+ 2384us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:53.038543 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:53.038548 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:53.038577 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:53.040424 (+ 1847us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.040998 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.041002 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.041003 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.041057 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.041352 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.041355 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.041356 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.041505 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:53.041600 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.042072 (+ 472us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.042204 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251025 08:02:53.042546 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 3979
I20251025 08:02:53.042659 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:02:53.042932 28176 external_mini_cluster.cc:1468] Setting key 822c9a3fd8181992d957efb34dcac893
I20251025 08:02:53.043066 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.030819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34111 (local address 127.27.132.1:32971)
1025 08:02:53.031271 (+ 452us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:53.031305 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:53.032087 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:53.032640 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:53.032648 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:53.033215 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:53.033980 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.033995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.035170 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.035174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:53.035826 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.035838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.036059 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.036743 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:53.036784 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:53.038370 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:53.040568 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.040574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.040586 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.040880 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.041149 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.041152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.041154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.041249 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.041616 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:53.041624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:53.041838 (+ 214us) client_negotiation.cc:769] Sending connection context
1025 08:02:53.042022 (+ 184us) client_negotiation.cc:240] Negotiation successful
1025 08:02:53.042279 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":142,"threads_started":1}
I20251025 08:02:53.044782 4110 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34111
I20251025 08:02:53.044950 4110 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:53.045531 4110 heartbeater.cc:507] Master 127.27.132.62:34111 requested a full tablet report, sending...
I20251025 08:02:53.047075 3904 ts_manager.cc:194] Registered new tserver with Master: 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1:43221)
I20251025 08:02:53.048761 3904 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:32971
I20251025 08:02:53.053447 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.046917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:39585 (local address 127.27.132.62:34111)
1025 08:02:53.047039 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.047044 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.047192 (+ 148us) server_negotiation.cc:408] Connection header received
1025 08:02:53.047312 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.047316 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.047391 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.047462 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:53.047975 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.048640 (+ 665us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.049279 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.049797 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.050426 (+ 629us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:53.050443 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:53.050446 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:53.050472 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:53.052045 (+ 1573us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.052476 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.052478 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.052479 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.052516 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.052766 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.052768 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.052769 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.052889 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:53.052957 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.053236 (+ 279us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.053333 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251025 08:02:53.053572 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.046861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34111 (local address 127.27.132.1:39585)
1025 08:02:53.047116 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:53.047129 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:53.047223 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:53.047481 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:53.047484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:53.047663 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:53.047859 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.047865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.048754 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.048758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:53.049173 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.049180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.049285 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.049824 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:53.049840 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:53.050266 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:53.052158 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.052162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.052164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.052383 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.052594 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.052599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.052604 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.052667 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.052978 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:53.052981 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:53.053079 (+ 98us) client_negotiation.cc:769] Sending connection context
1025 08:02:53.053210 (+ 131us) client_negotiation.cc:240] Negotiation successful
1025 08:02:53.053366 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for oryx/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
I20251025 08:02:53.088124 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34111
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:53.200979 4121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:53.201246 4121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:53.201339 4121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:53.204649 4121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:53.204723 4121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:53.204783 4121 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 08:02:53.204862 4121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:02:53.209182 4121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.132.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:53.210325 4121 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:53.211258 4121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:53.218046 4129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:53.218025 4121 server_base.cc:1047] running on GCE node
W20251025 08:02:53.217970 4126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:53.217970 4127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:53.218597 4121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:53.219144 4121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:53.220304 4121 hybrid_clock.cc:648] HybridClock initialized: now 1761379373220288 us; error 33 us; skew 500 ppm
Oct 25 08:02:53 dist-test-slave-lq8l krb5kdc[3857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379373, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:53.222923 4121 init.cc:377] Logged in from keytab as oryx/127.27.132.2@KRBTEST.COM (short username oryx)
I20251025 08:02:53.224112 4121 webserver.cc:492] Webserver started at http://127.27.132.2:33375/ using document root <none> and password file <none>
I20251025 08:02:53.224640 4121 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:53.224689 4121 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:53.224851 4121 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:53.226557 4121 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "abebaf13080d42eb9b7574bd956fb9ef"
format_stamp: "Formatted at 2025-10-25 08:02:53 on dist-test-slave-lq8l"
server_key: "5a60aa5896a836e863103b7fc135b69e"
server_key_iv: "8e6f4de80e61d52bc23deff822ac6b4d"
server_key_version: "encryptionkey@0"
I20251025 08:02:53.226999 4121 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "abebaf13080d42eb9b7574bd956fb9ef"
format_stamp: "Formatted at 2025-10-25 08:02:53 on dist-test-slave-lq8l"
server_key: "5a60aa5896a836e863103b7fc135b69e"
server_key_iv: "8e6f4de80e61d52bc23deff822ac6b4d"
server_key_version: "encryptionkey@0"
I20251025 08:02:53.230513 4121 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251025 08:02:53.233003 4136 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:53.234135 4121 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251025 08:02:53.234239 4121 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "abebaf13080d42eb9b7574bd956fb9ef"
format_stamp: "Formatted at 2025-10-25 08:02:53 on dist-test-slave-lq8l"
server_key: "5a60aa5896a836e863103b7fc135b69e"
server_key_iv: "8e6f4de80e61d52bc23deff822ac6b4d"
server_key_version: "encryptionkey@0"
I20251025 08:02:53.234328 4121 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:53.251916 4121 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:53.254884 4121 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:53.255048 4121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:53.256596 4121 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:53.256656 4121 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:53.256707 4121 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:53.256796 4121 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Oct 25 08:02:53 dist-test-slave-lq8l krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379373, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:53.268304 4121 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:44491
I20251025 08:02:53.268333 4251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:44491 every 8 connection(s)
I20251025 08:02:53.269380 4121 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:02:53.269555 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.257248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:40615 (local address 127.27.132.62:34111)
1025 08:02:53.257376 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.257380 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.258229 (+ 849us) server_negotiation.cc:408] Connection header received
1025 08:02:53.259138 (+ 909us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.259142 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.259210 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.259292 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:53.260895 (+ 1603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.261553 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.262208 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.262395 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.265387 (+ 2992us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:53.265412 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:53.265415 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:53.265448 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:53.267317 (+ 1869us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.267983 (+ 666us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.267986 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.267988 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.268051 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.268398 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.268401 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.268402 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.268573 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:53.268676 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.269203 (+ 527us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.269381 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251025 08:02:53.270287 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.257615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34111 (local address 127.27.132.2:40615)
1025 08:02:53.258086 (+ 471us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:53.258122 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:53.258921 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:53.259499 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:53.259508 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:53.260099 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:53.260741 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.260755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.261676 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.261680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:53.262091 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.262099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.262324 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.263071 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:53.263108 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:53.265221 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:53.267477 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.267483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.267501 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.267863 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.268174 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.268178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.268181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.268291 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.268686 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:53.268695 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:53.268941 (+ 246us) client_negotiation.cc:769] Sending connection context
1025 08:02:53.269148 (+ 207us) client_negotiation.cc:240] Negotiation successful
1025 08:02:53.269450 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":133,"threads_started":1}
I20251025 08:02:53.271659 4252 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34111
I20251025 08:02:53.271863 4252 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:53.272437 4252 heartbeater.cc:507] Master 127.27.132.62:34111 requested a full tablet report, sending...
I20251025 08:02:53.273651 3905 ts_manager.cc:194] Registered new tserver with Master: abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2:44491)
I20251025 08:02:53.274016 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 4121
I20251025 08:02:53.274118 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:02:53.274364 28176 external_mini_cluster.cc:1468] Setting key 704a8072bc821cc2493a1155eb1f9cb4
I20251025 08:02:53.274477 3905 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:40615
I20251025 08:02:53.281332 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.273995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:51471 (local address 127.27.132.62:34111)
1025 08:02:53.274138 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.274144 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.274538 (+ 394us) server_negotiation.cc:408] Connection header received
1025 08:02:53.274659 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.274663 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.274715 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.274788 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:53.275576 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.276061 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.276666 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.276890 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.277898 (+ 1008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:53.277913 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:53.277915 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:53.277939 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:53.279624 (+ 1685us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.280138 (+ 514us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.280141 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.280143 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.280193 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.280500 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.280503 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.280504 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.280710 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:53.280793 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.281109 (+ 316us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.281203 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251025 08:02:53.281530 4258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.273943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34111 (local address 127.27.132.2:51471)
1025 08:02:53.274448 (+ 505us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:53.274461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:53.274566 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:53.274912 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:53.274915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:53.275133 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:53.275439 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.275449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.276170 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.276173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:53.276558 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.276566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.276674 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.277330 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:53.277344 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:53.277761 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:53.279756 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.279762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.279766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.280009 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.280299 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.280303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.280305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.280379 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.280811 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:53.280817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:53.280939 (+ 122us) client_negotiation.cc:769] Sending connection context
1025 08:02:53.281077 (+ 138us) client_negotiation.cc:240] Negotiation successful
1025 08:02:53.281249 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for oryx/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab.
I20251025 08:02:53.322723 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34111
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:53.432310 4263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:53.432585 4263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:53.432704 4263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:53.436038 4263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:53.436122 4263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:53.436220 4263 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251025 08:02:53.436301 4263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:02:53.440778 4263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.132.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:53.441938 4263 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:53.442806 4263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:53.449738 4271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:53.449793 4268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:53.449865 4263 server_base.cc:1047] running on GCE node
W20251025 08:02:53.449729 4269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:53.450419 4263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:53.450999 4263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:53.452142 4263 hybrid_clock.cc:648] HybridClock initialized: now 1761379373452119 us; error 39 us; skew 500 ppm
Oct 25 08:02:53 dist-test-slave-lq8l krb5kdc[3857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379373, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:53.454918 4263 init.cc:377] Logged in from keytab as oryx/127.27.132.3@KRBTEST.COM (short username oryx)
I20251025 08:02:53.456267 4263 webserver.cc:492] Webserver started at http://127.27.132.3:36931/ using document root <none> and password file <none>
I20251025 08:02:53.457005 4263 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:53.457063 4263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:53.457239 4263 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:53.459018 4263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7"
format_stamp: "Formatted at 2025-10-25 08:02:53 on dist-test-slave-lq8l"
server_key: "1f2c5d902e23f300ad2ea0bda4c54cd1"
server_key_iv: "8d8a8aebbf959eadfe3be36067999067"
server_key_version: "encryptionkey@0"
I20251025 08:02:53.459519 4263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7"
format_stamp: "Formatted at 2025-10-25 08:02:53 on dist-test-slave-lq8l"
server_key: "1f2c5d902e23f300ad2ea0bda4c54cd1"
server_key_iv: "8d8a8aebbf959eadfe3be36067999067"
server_key_version: "encryptionkey@0"
I20251025 08:02:53.463047 4263 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251025 08:02:53.465420 4278 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:53.466511 4263 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251025 08:02:53.466626 4263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7"
format_stamp: "Formatted at 2025-10-25 08:02:53 on dist-test-slave-lq8l"
server_key: "1f2c5d902e23f300ad2ea0bda4c54cd1"
server_key_iv: "8d8a8aebbf959eadfe3be36067999067"
server_key_version: "encryptionkey@0"
I20251025 08:02:53.466720 4263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:53.505095 4263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:53.507911 4263 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:53.508105 4263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:53.509618 4263 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:02:53.509671 4263 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:53.509737 4263 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:02:53.509766 4263 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 25 08:02:53 dist-test-slave-lq8l krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379373, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:53.520565 4263 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:40113
I20251025 08:02:53.520594 4393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:40113 every 8 connection(s)
I20251025 08:02:53.521517 4263 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:02:53.522964 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.510655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34111 (local address 127.27.132.3:32995)
1025 08:02:53.511094 (+ 439us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:53.511136 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:53.512004 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:53.512587 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:53.512599 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:53.513192 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:53.513944 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.513964 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.515125 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.515129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:53.515691 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.515699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.515933 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.516626 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:53.516667 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:53.518457 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:53.520612 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.520618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.520631 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.520906 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.521145 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.521150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.521152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.521229 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.521595 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:53.521604 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:53.521836 (+ 232us) client_negotiation.cc:769] Sending connection context
1025 08:02:53.522043 (+ 207us) client_negotiation.cc:240] Negotiation successful
1025 08:02:53.522296 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
I20251025 08:02:53.523573 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.510309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:32995 (local address 127.27.132.62:34111)
1025 08:02:53.510469 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.510474 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.511255 (+ 781us) server_negotiation.cc:408] Connection header received
1025 08:02:53.512217 (+ 962us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.512221 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.512293 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.512394 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:53.514140 (+ 1746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.514993 (+ 853us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.515814 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.516004 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.518613 (+ 2609us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:53.518641 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:53.518647 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:53.518686 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:53.520496 (+ 1810us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.521006 (+ 510us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.521009 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.521011 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.521062 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.521316 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.521319 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.521321 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.521486 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:53.521586 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.523305 (+ 1719us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.523458 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 08:02:53.524554 4394 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34111
I20251025 08:02:53.524718 4394 heartbeater.cc:461] Registering TS with master...
I20251025 08:02:53.525337 4394 heartbeater.cc:507] Master 127.27.132.62:34111 requested a full tablet report, sending...
I20251025 08:02:53.526340 3904 ts_manager.cc:194] Registered new tserver with Master: 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113)
I20251025 08:02:53.527089 3904 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:32995
I20251025 08:02:53.529711 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 4263
I20251025 08:02:53.529808 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:02:53.530046 28176 external_mini_cluster.cc:1468] Setting key 350677ba0409d92a87048a978eef66fb
I20251025 08:02:53.532398 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 08:02:53.533556 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.526558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34111 (local address 127.27.132.3:42567)
1025 08:02:53.526820 (+ 262us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:53.526837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:53.526929 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:53.527237 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:53.527243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:53.527452 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:53.527726 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.527734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.528423 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.528427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:53.528960 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.528970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.529085 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.529662 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:53.529677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:53.530025 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:53.532331 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.532336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.532339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.532520 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.532763 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:53.532766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:53.532767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:53.532808 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:53.533159 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:53.533161 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:53.533236 (+ 75us) client_negotiation.cc:769] Sending connection context
1025 08:02:53.533311 (+ 75us) client_negotiation.cc:240] Negotiation successful
1025 08:02:53.533416 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":108,"threads_started":1}
I20251025 08:02:53.533742 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.526905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:42567 (local address 127.27.132.62:34111)
1025 08:02:53.527027 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.527031 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.527044 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:02:53.527080 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.527084 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.527130 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.527213 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:53.527847 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.528317 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.529092 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.529244 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.530146 (+ 902us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:53.530163 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:53.530166 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:53.530193 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:53.532192 (+ 1999us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.532616 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.532620 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.532622 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.532674 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.532880 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.532884 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.532887 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.533069 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:53.533173 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.533378 (+ 205us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.533518 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Oct 25 08:02:53 dist-test-slave-lq8l krb5kdc[3857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 25 08:02:53 dist-test-slave-lq8l krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:02:53 dist-test-slave-lq8l krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
W20251025 08:02:53.554258 3954 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:53.549007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.27.132.62:34111)
1025 08:02:53.549125 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.549128 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.549141 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:02:53.549181 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.549184 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.549226 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.549300 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:53.550054 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.550554 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.551152 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.551270 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.554104 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51270: BlockingRecv error: recv got EOF from 127.0.0.1:51270 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Oct 25 08:02:53 dist-test-slave-lq8l krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:53.562903 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.555715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.27.132.62:34111)
1025 08:02:53.555856 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.555861 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.555943 (+ 82us) server_negotiation.cc:408] Connection header received
1025 08:02:53.556082 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.556116 (+ 34us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.556164 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.556248 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:53.557159 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.557626 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.558213 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.558415 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.560332 (+ 1917us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:53.560347 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:53.560350 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:53.560373 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:53.561545 (+ 1172us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.561907 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.561910 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.561911 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.561952 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:53.562150 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:53.562153 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:53.562155 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:53.562293 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:53.562414 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.562624 (+ 210us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.562734 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251025 08:02:53.565332 3904 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:02:53.567636 3904 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:02:53.580127 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.575462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:44491 (local address 127.0.0.1:37746)
1025 08:02:53.576301 (+ 839us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:53.576315 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:53.576386 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:53.577669 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:53.577672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:53.577688 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:53.577855 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.577860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.578961 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.578964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:53.579723 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.579732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.579812 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.579853 (+ 41us) client_negotiation.cc:769] Sending connection context
1025 08:02:53.579912 (+ 59us) client_negotiation.cc:240] Negotiation successful
1025 08:02:53.579980 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":111,"threads_started":1}
I20251025 08:02:53.580629 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.575270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:43221 (local address 127.0.0.1:33978)
1025 08:02:53.575885 (+ 615us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:53.575927 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:53.576055 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:53.577085 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:53.577091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:53.577127 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:53.577485 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.577494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.578941 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.578945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:53.580170 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.580177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.580377 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.580394 (+ 17us) client_negotiation.cc:769] Sending connection context
1025 08:02:53.580441 (+ 47us) client_negotiation.cc:240] Negotiation successful
1025 08:02:53.580492 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"spinlock_wait_cycles":4992,"thread_start_us":120,"threads_started":1}
I20251025 08:02:53.580530 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.575971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:40113 (local address 127.0.0.1:41506)
1025 08:02:53.576406 (+ 435us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:53.576424 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:53.576524 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:53.577085 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:53.577091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:53.577128 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:53.577485 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.577494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.579069 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.579072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:53.579965 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.579975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.580085 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.580100 (+ 15us) client_negotiation.cc:769] Sending connection context
1025 08:02:53.580148 (+ 48us) client_negotiation.cc:240] Negotiation successful
1025 08:02:53.580210 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"spinlock_wait_cycles":6912,"thread_start_us":90,"threads_started":1}
I20251025 08:02:53.581104 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.576347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.27.132.3:40113)
1025 08:02:53.576626 (+ 279us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.576633 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.576656 (+ 23us) server_negotiation.cc:408] Connection header received
1025 08:02:53.576744 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.576748 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.576930 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.577071 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:53.577874 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.578962 (+ 1088us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.580120 (+ 1158us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.580703 (+ 583us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.580819 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.580886 (+ 67us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.580972 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20251025 08:02:53.581686 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.575460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33978 (local address 127.27.132.1:43221)
1025 08:02:53.576626 (+ 1166us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.576633 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.576656 (+ 23us) server_negotiation.cc:408] Connection header received
1025 08:02:53.576740 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.576746 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.576929 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.577071 (+ 142us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:53.577627 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.578800 (+ 1173us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.580600 (+ 1800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.581400 (+ 800us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.581477 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.581529 (+ 52us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.581593 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":79,"threads_started":1}
I20251025 08:02:53.582017 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.576024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.27.132.2:44491)
1025 08:02:53.577340 (+ 1316us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.577345 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.577362 (+ 17us) server_negotiation.cc:408] Connection header received
1025 08:02:53.577425 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.577428 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.577550 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.577693 (+ 143us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:53.577967 (+ 274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.578799 (+ 832us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.580513 (+ 1714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.581022 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.581118 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.581186 (+ 68us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.581870 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225,"thread_start_us":136,"threads_started":1}
I20251025 08:02:53.584223 4186 tablet_service.cc:1505] Processing CreateTablet for tablet 561866b43a0048748cc37c3bf5bc45a7 (DEFAULT_TABLE table=test-table [id=72daaa16dd484db1b6bfb87d99233652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:53.584121 4040 tablet_service.cc:1505] Processing CreateTablet for tablet 561866b43a0048748cc37c3bf5bc45a7 (DEFAULT_TABLE table=test-table [id=72daaa16dd484db1b6bfb87d99233652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:53.584132 4328 tablet_service.cc:1505] Processing CreateTablet for tablet 561866b43a0048748cc37c3bf5bc45a7 (DEFAULT_TABLE table=test-table [id=72daaa16dd484db1b6bfb87d99233652]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:53.584985 4186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 561866b43a0048748cc37c3bf5bc45a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:53.585012 4040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 561866b43a0048748cc37c3bf5bc45a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:53.585012 4328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 561866b43a0048748cc37c3bf5bc45a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:53.591034 4422 tablet_bootstrap.cc:492] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef: Bootstrap starting.
I20251025 08:02:53.591800 4424 tablet_bootstrap.cc:492] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Bootstrap starting.
I20251025 08:02:53.593043 4422 tablet_bootstrap.cc:654] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:53.593415 4424 tablet_bootstrap.cc:654] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:53.594077 4422 log.cc:826] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:53.594364 4424 log.cc:826] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:53.594991 4423 tablet_bootstrap.cc:492] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5: Bootstrap starting.
I20251025 08:02:53.595916 4422 tablet_bootstrap.cc:492] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef: No bootstrap required, opened a new log
I20251025 08:02:53.596105 4422 ts_tablet_manager.cc:1403] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 08:02:53.596145 4424 tablet_bootstrap.cc:492] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: No bootstrap required, opened a new log
I20251025 08:02:53.596310 4424 ts_tablet_manager.cc:1403] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 08:02:53.597211 4423 tablet_bootstrap.cc:654] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:53.597951 4423 log.cc:826] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:53.599257 4422 raft_consensus.cc:359] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.599259 4424 raft_consensus.cc:359] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.599510 4424 raft_consensus.cc:385] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:53.599519 4422 raft_consensus.cc:385] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:53.599568 4422 raft_consensus.cc:740] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abebaf13080d42eb9b7574bd956fb9ef, State: Initialized, Role: FOLLOWER
I20251025 08:02:53.599568 4424 raft_consensus.cc:740] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2e51eb82624cc4a3f5a8f51ceabfb7, State: Initialized, Role: FOLLOWER
I20251025 08:02:53.599752 4423 tablet_bootstrap.cc:492] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5: No bootstrap required, opened a new log
I20251025 08:02:53.599933 4423 ts_tablet_manager.cc:1403] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 08:02:53.600072 4422 consensus_queue.cc:260] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.600072 4424 consensus_queue.cc:260] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.601059 4424 ts_tablet_manager.cc:1434] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251025 08:02:53.601229 4394 heartbeater.cc:499] Master 127.27.132.62:34111 was elected leader, sending a full tablet report...
I20251025 08:02:53.601059 4422 ts_tablet_manager.cc:1434] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251025 08:02:53.601296 4252 heartbeater.cc:499] Master 127.27.132.62:34111 was elected leader, sending a full tablet report...
I20251025 08:02:53.602845 4423 raft_consensus.cc:359] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.603070 4423 raft_consensus.cc:385] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:53.603142 4423 raft_consensus.cc:740] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2826ef6b198d4fe79b769f1450049ec5, State: Initialized, Role: FOLLOWER
I20251025 08:02:53.603633 4423 consensus_queue.cc:260] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.604404 4110 heartbeater.cc:499] Master 127.27.132.62:34111 was elected leader, sending a full tablet report...
I20251025 08:02:53.604553 4423 ts_tablet_manager.cc:1434] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20251025 08:02:53.704007 4430 raft_consensus.cc:493] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:02:53.704198 4430 raft_consensus.cc:515] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.705314 4430 leader_election.cc:290] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2:44491), 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113)
I20251025 08:02:53.709010 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.705600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:44491 (local address 127.27.132.1:36981)
1025 08:02:53.706039 (+ 439us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:53.706051 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:53.706154 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:53.706445 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:53.706450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:53.706474 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:53.706794 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.706801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.707754 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.707757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:53.708559 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.708568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.708689 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.708710 (+ 21us) client_negotiation.cc:769] Sending connection context
1025 08:02:53.708763 (+ 53us) client_negotiation.cc:240] Negotiation successful
1025 08:02:53.708816 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":684,"threads_started":1}
I20251025 08:02:53.709185 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.705615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:40113 (local address 127.27.132.1:49275)
1025 08:02:53.706441 (+ 826us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:53.706454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:53.706552 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:53.706796 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:53.706799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:53.706816 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:53.707066 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.707072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.708016 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.708019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:53.708844 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:53.708851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.708947 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.708960 (+ 13us) client_negotiation.cc:769] Sending connection context
1025 08:02:53.709007 (+ 47us) client_negotiation.cc:240] Negotiation successful
1025 08:02:53.709060 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":147,"threads_started":1}
I20251025 08:02:53.709364 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.705702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:36981 (local address 127.27.132.2:44491)
1025 08:02:53.705942 (+ 240us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.705949 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.706130 (+ 181us) server_negotiation.cc:408] Connection header received
1025 08:02:53.706246 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.706252 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.706317 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.706414 (+ 97us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:53.706899 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.707624 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.708684 (+ 1060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.709092 (+ 408us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.709133 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.709182 (+ 49us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.709239 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20251025 08:02:53.709686 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.705742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:49275 (local address 127.27.132.3:40113)
1025 08:02:53.705918 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.705923 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.706526 (+ 603us) server_negotiation.cc:408] Connection header received
1025 08:02:53.706649 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.706651 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.706700 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.706760 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:53.707185 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.707912 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.708966 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.709404 (+ 438us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.709454 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.709505 (+ 51us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.709566 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251025 08:02:53.710083 4206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "561866b43a0048748cc37c3bf5bc45a7" candidate_uuid: "2826ef6b198d4fe79b769f1450049ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abebaf13080d42eb9b7574bd956fb9ef" is_pre_election: true
I20251025 08:02:53.710227 4348 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "561866b43a0048748cc37c3bf5bc45a7" candidate_uuid: "2826ef6b198d4fe79b769f1450049ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" is_pre_election: true
I20251025 08:02:53.710390 4206 raft_consensus.cc:2468] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2826ef6b198d4fe79b769f1450049ec5 in term 0.
I20251025 08:02:53.710434 4348 raft_consensus.cc:2468] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2826ef6b198d4fe79b769f1450049ec5 in term 0.
I20251025 08:02:53.710927 3998 leader_election.cc:304] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2826ef6b198d4fe79b769f1450049ec5, abebaf13080d42eb9b7574bd956fb9ef; no voters:
I20251025 08:02:53.711179 4430 raft_consensus.cc:2804] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:02:53.711254 4430 raft_consensus.cc:493] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:02:53.711303 4430 raft_consensus.cc:3060] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:53.712255 4430 raft_consensus.cc:515] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.712610 4430 leader_election.cc:290] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [CANDIDATE]: Term 1 election: Requested vote from peers abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2:44491), 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113)
I20251025 08:02:53.712950 4206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "561866b43a0048748cc37c3bf5bc45a7" candidate_uuid: "2826ef6b198d4fe79b769f1450049ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abebaf13080d42eb9b7574bd956fb9ef"
I20251025 08:02:53.712976 4348 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "561866b43a0048748cc37c3bf5bc45a7" candidate_uuid: "2826ef6b198d4fe79b769f1450049ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7"
I20251025 08:02:53.713078 4206 raft_consensus.cc:3060] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:53.713078 4348 raft_consensus.cc:3060] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:53.714294 4348 raft_consensus.cc:2468] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2826ef6b198d4fe79b769f1450049ec5 in term 1.
I20251025 08:02:53.714294 4206 raft_consensus.cc:2468] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2826ef6b198d4fe79b769f1450049ec5 in term 1.
I20251025 08:02:53.714627 3997 leader_election.cc:304] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2826ef6b198d4fe79b769f1450049ec5, 6b2e51eb82624cc4a3f5a8f51ceabfb7; no voters:
I20251025 08:02:53.714794 4430 raft_consensus.cc:2804] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:53.714999 4430 raft_consensus.cc:697] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 1 LEADER]: Becoming Leader. State: Replica: 2826ef6b198d4fe79b769f1450049ec5, State: Running, Role: LEADER
I20251025 08:02:53.715293 4430 consensus_queue.cc:237] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.718561 3904 catalog_manager.cc:5649] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1). New cstate: current_term: 1 leader_uuid: "2826ef6b198d4fe79b769f1450049ec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20251025 08:02:53.733101 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:53.729984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.27.132.1:43221)
1025 08:02:53.730114 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:53.730118 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:53.730130 (+ 12us) server_negotiation.cc:408] Connection header received
1025 08:02:53.730205 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:53.730207 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:53.730261 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:53.730342 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:02:53.730638 (+ 296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.731131 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:53.732074 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:53.732213 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:53.732264 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:02:53.732746 (+ 482us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:02:53.732831 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:53.732913 (+ 82us) server_negotiation.cc:300] Negotiation successful
1025 08:02:53.732957 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 08:02:53.738692 4348 raft_consensus.cc:1275] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Refusing update from remote peer 2826ef6b198d4fe79b769f1450049ec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:53.738692 4206 raft_consensus.cc:1275] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Refusing update from remote peer 2826ef6b198d4fe79b769f1450049ec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:53.739476 4433 consensus_queue.cc:1048] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:53.739681 4430 consensus_queue.cc:1048] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:53.746443 4436 mvcc.cc:204] Tried to move back new op lower bound from 7214609914826547200 to 7214609914740862976. Current Snapshot: MvccSnapshot[applied={T|T < 7214609914826547200}]
I20251025 08:02:53.749565 4437 mvcc.cc:204] Tried to move back new op lower bound from 7214609914826547200 to 7214609914740862976. Current Snapshot: MvccSnapshot[applied={T|T < 7214609914826547200 or (T in {7214609914826547200})}]
I20251025 08:02:53.755067 3903 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51274:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:53.755235 3903 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51274:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:53.757851 3903 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3: Sending DeleteTablet for 3 replicas of tablet 561866b43a0048748cc37c3bf5bc45a7
I20251025 08:02:53.758752 4186 tablet_service.cc:1552] Processing DeleteTablet for tablet 561866b43a0048748cc37c3bf5bc45a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:53 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:37746
I20251025 08:02:53.758868 4040 tablet_service.cc:1552] Processing DeleteTablet for tablet 561866b43a0048748cc37c3bf5bc45a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:53 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:33978
I20251025 08:02:53.759366 4328 tablet_service.cc:1552] Processing DeleteTablet for tablet 561866b43a0048748cc37c3bf5bc45a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:53 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:41506
I20251025 08:02:53.759469 3903 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:02:53.759877 3903 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:02:53.760133 4447 tablet_replica.cc:333] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: stopping tablet replica
I20251025 08:02:53.760473 4447 raft_consensus.cc:2243] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:53.760906 4447 raft_consensus.cc:2272] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:53.761355 4446 tablet_replica.cc:333] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5: stopping tablet replica
I20251025 08:02:53.761583 4446 raft_consensus.cc:2243] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:02:53.761942 4446 raft_consensus.cc:2272] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:53.765067 4186 tablet_service.cc:1505] Processing CreateTablet for tablet 1ad7078ed28b49829d29038d5b7fbe1e (DEFAULT_TABLE table=test-table [id=da02e77b51c5455089836aa13fdcefe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:53.765352 4186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad7078ed28b49829d29038d5b7fbe1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:53.765702 4328 tablet_service.cc:1505] Processing CreateTablet for tablet 1ad7078ed28b49829d29038d5b7fbe1e (DEFAULT_TABLE table=test-table [id=da02e77b51c5455089836aa13fdcefe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:53.765974 4328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad7078ed28b49829d29038d5b7fbe1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:53.766156 4445 tablet_replica.cc:333] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef: stopping tablet replica
I20251025 08:02:53.766400 4445 raft_consensus.cc:2243] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:53.766750 4445 raft_consensus.cc:2272] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:53.767982 4422 tablet_bootstrap.cc:492] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef: Bootstrap starting.
I20251025 08:02:53.768196 4445 ts_tablet_manager.cc:1916] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:53.768996 4422 tablet_bootstrap.cc:654] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:53.769433 4424 tablet_bootstrap.cc:492] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Bootstrap starting.
I20251025 08:02:53.770542 4424 tablet_bootstrap.cc:654] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:53.765451 4040 tablet_service.cc:1505] Processing CreateTablet for tablet 1ad7078ed28b49829d29038d5b7fbe1e (DEFAULT_TABLE table=test-table [id=da02e77b51c5455089836aa13fdcefe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:53.770953 4040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad7078ed28b49829d29038d5b7fbe1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:53.771087 4447 ts_tablet_manager.cc:1916] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:53.772778 4445 ts_tablet_manager.cc:1929] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:53.772874 4445 log.cc:1199] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/561866b43a0048748cc37c3bf5bc45a7
I20251025 08:02:53.773211 4445 ts_tablet_manager.cc:1950] T 561866b43a0048748cc37c3bf5bc45a7 P abebaf13080d42eb9b7574bd956fb9ef: Deleting consensus metadata
I20251025 08:02:53.773561 4423 tablet_bootstrap.cc:492] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5: Bootstrap starting.
I20251025 08:02:53.764532 4446 ts_tablet_manager.cc:1916] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:53.774602 3892 catalog_manager.cc:4985] TS abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2:44491): tablet 561866b43a0048748cc37c3bf5bc45a7 (table test-table [id=72daaa16dd484db1b6bfb87d99233652]) successfully deleted
I20251025 08:02:53.774672 4423 tablet_bootstrap.cc:654] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:53.775421 4422 tablet_bootstrap.cc:492] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef: No bootstrap required, opened a new log
I20251025 08:02:53.775509 4422 ts_tablet_manager.cc:1403] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20251025 08:02:53.775712 4424 tablet_bootstrap.cc:492] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7: No bootstrap required, opened a new log
I20251025 08:02:53.775801 4424 ts_tablet_manager.cc:1403] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20251025 08:02:53.775959 4422 raft_consensus.cc:359] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.776062 4422 raft_consensus.cc:385] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:53.776093 4422 raft_consensus.cc:740] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abebaf13080d42eb9b7574bd956fb9ef, State: Initialized, Role: FOLLOWER
I20251025 08:02:53.776213 4422 consensus_queue.cc:260] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.776280 4424 raft_consensus.cc:359] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.776387 4424 raft_consensus.cc:385] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:53.776417 4424 raft_consensus.cc:740] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2e51eb82624cc4a3f5a8f51ceabfb7, State: Initialized, Role: FOLLOWER
I20251025 08:02:53.776544 4424 consensus_queue.cc:260] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.776809 4424 ts_tablet_manager.cc:1434] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251025 08:02:53.776991 4446 ts_tablet_manager.cc:1929] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:53.777074 4446 log.cc:1199] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/wal/wals/561866b43a0048748cc37c3bf5bc45a7
I20251025 08:02:53.777465 4422 ts_tablet_manager.cc:1434] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:53.778252 4446 ts_tablet_manager.cc:1950] T 561866b43a0048748cc37c3bf5bc45a7 P 2826ef6b198d4fe79b769f1450049ec5: Deleting consensus metadata
I20251025 08:02:53.778837 4447 ts_tablet_manager.cc:1929] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:53.778937 4447 log.cc:1199] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/561866b43a0048748cc37c3bf5bc45a7
I20251025 08:02:53.779486 3891 catalog_manager.cc:4985] TS 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1:43221): tablet 561866b43a0048748cc37c3bf5bc45a7 (table test-table [id=72daaa16dd484db1b6bfb87d99233652]) successfully deleted
I20251025 08:02:53.779893 4447 ts_tablet_manager.cc:1950] T 561866b43a0048748cc37c3bf5bc45a7 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting consensus metadata
I20251025 08:02:53.781173 3891 catalog_manager.cc:4985] TS 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113): tablet 561866b43a0048748cc37c3bf5bc45a7 (table test-table [id=72daaa16dd484db1b6bfb87d99233652]) successfully deleted
I20251025 08:02:53.784116 4423 tablet_bootstrap.cc:492] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5: No bootstrap required, opened a new log
I20251025 08:02:53.784204 4423 ts_tablet_manager.cc:1403] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20251025 08:02:53.784619 4423 raft_consensus.cc:359] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.784718 4423 raft_consensus.cc:385] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:53.784771 4423 raft_consensus.cc:740] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2826ef6b198d4fe79b769f1450049ec5, State: Initialized, Role: FOLLOWER
I20251025 08:02:53.784911 4423 consensus_queue.cc:260] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:53.785153 4423 ts_tablet_manager.cc:1434] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20251025 08:02:53.793656 4111 tablet.cc:2378] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251025 08:02:54.020778 4253 tablet.cc:2378] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251025 08:02:54.022475 4395 tablet.cc:2378] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 08:02:54.025596 4430 raft_consensus.cc:493] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:02:54.025707 4430 raft_consensus.cc:515] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.026160 4430 leader_election.cc:290] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2:44491), 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113)
I20251025 08:02:54.026604 4206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ad7078ed28b49829d29038d5b7fbe1e" candidate_uuid: "2826ef6b198d4fe79b769f1450049ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abebaf13080d42eb9b7574bd956fb9ef" is_pre_election: true
I20251025 08:02:54.026633 4348 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ad7078ed28b49829d29038d5b7fbe1e" candidate_uuid: "2826ef6b198d4fe79b769f1450049ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" is_pre_election: true
I20251025 08:02:54.026757 4206 raft_consensus.cc:2468] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2826ef6b198d4fe79b769f1450049ec5 in term 0.
I20251025 08:02:54.026752 4348 raft_consensus.cc:2468] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2826ef6b198d4fe79b769f1450049ec5 in term 0.
I20251025 08:02:54.027065 3997 leader_election.cc:304] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2826ef6b198d4fe79b769f1450049ec5, 6b2e51eb82624cc4a3f5a8f51ceabfb7; no voters:
I20251025 08:02:54.027240 4430 raft_consensus.cc:2804] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:02:54.027303 4430 raft_consensus.cc:493] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:02:54.027390 4430 raft_consensus.cc:3060] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:54.028275 4430 raft_consensus.cc:515] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.028590 4430 leader_election.cc:290] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [CANDIDATE]: Term 1 election: Requested vote from peers abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2:44491), 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113)
I20251025 08:02:54.028945 4206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ad7078ed28b49829d29038d5b7fbe1e" candidate_uuid: "2826ef6b198d4fe79b769f1450049ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abebaf13080d42eb9b7574bd956fb9ef"
I20251025 08:02:54.028959 4348 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ad7078ed28b49829d29038d5b7fbe1e" candidate_uuid: "2826ef6b198d4fe79b769f1450049ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7"
I20251025 08:02:54.029043 4206 raft_consensus.cc:3060] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:54.029065 4348 raft_consensus.cc:3060] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:54.029858 4206 raft_consensus.cc:2468] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2826ef6b198d4fe79b769f1450049ec5 in term 1.
I20251025 08:02:54.029903 4348 raft_consensus.cc:2468] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2826ef6b198d4fe79b769f1450049ec5 in term 1.
I20251025 08:02:54.030150 3998 leader_election.cc:304] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2826ef6b198d4fe79b769f1450049ec5, abebaf13080d42eb9b7574bd956fb9ef; no voters:
I20251025 08:02:54.030323 4430 raft_consensus.cc:2804] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:54.030418 4430 raft_consensus.cc:697] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 1 LEADER]: Becoming Leader. State: Replica: 2826ef6b198d4fe79b769f1450049ec5, State: Running, Role: LEADER
I20251025 08:02:54.030578 4430 consensus_queue.cc:237] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.032168 3905 catalog_manager.cc:5649] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1). New cstate: current_term: 1 leader_uuid: "2826ef6b198d4fe79b769f1450049ec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } health_report { overall_health: UNKNOWN } } }
Oct 25 08:02:54 dist-test-slave-lq8l krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379372, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.62@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:54.102017 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.093217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51286 (local address 127.27.132.62:34111)
1025 08:02:54.093492 (+ 275us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:54.093496 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:54.093512 (+ 16us) server_negotiation.cc:408] Connection header received
1025 08:02:54.093666 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:54.093670 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:54.093719 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:54.093815 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:54.094628 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.095316 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.096186 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.096356 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.098771 (+ 2415us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:54.098793 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:54.098797 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:54.098825 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:54.100324 (+ 1499us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:54.100803 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:54.100806 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:54.100807 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:54.100842 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:54.101014 (+ 172us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:54.101016 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:54.101017 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:54.101164 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:54.101235 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:54.101611 (+ 376us) server_negotiation.cc:300] Negotiation successful
1025 08:02:54.101849 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
I20251025 08:02:54.102016 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.093140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34111 (local address 127.0.0.1:51286)
1025 08:02:54.093435 (+ 295us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:54.093450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:54.093548 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:54.093849 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:54.093851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:54.094285 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:02:54.094478 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.094485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.095469 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.095473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:54.096071 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.096084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.096205 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.096861 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:02:54.096880 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:02:54.098604 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:02:54.100432 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:54.100440 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:54.100445 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:54.100721 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:54.100905 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:02:54.100907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:02:54.100909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:02:54.100957 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:02:54.101286 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:02:54.101297 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:02:54.101405 (+ 108us) client_negotiation.cc:769] Sending connection context
1025 08:02:54.101521 (+ 116us) client_negotiation.cc:240] Negotiation successful
1025 08:02:54.101746 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
I20251025 08:02:54.106616 3905 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:51286:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251025 08:02:54.107396 3905 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:02:54.112252 4040 tablet_service.cc:1505] Processing CreateTablet for tablet 3c7cf5a99e5548a39d5f142562536222 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63c7a7618f4447ef9b610fbc4a185d7e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 08:02:54.112452 4186 tablet_service.cc:1505] Processing CreateTablet for tablet 3c7cf5a99e5548a39d5f142562536222 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63c7a7618f4447ef9b610fbc4a185d7e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 08:02:54.112694 4040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c7cf5a99e5548a39d5f142562536222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:54.112700 4328 tablet_service.cc:1505] Processing CreateTablet for tablet 3c7cf5a99e5548a39d5f142562536222 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63c7a7618f4447ef9b610fbc4a185d7e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251025 08:02:54.112833 4186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c7cf5a99e5548a39d5f142562536222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:54.113034 4328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c7cf5a99e5548a39d5f142562536222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:54.115217 4422 tablet_bootstrap.cc:492] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: Bootstrap starting.
I20251025 08:02:54.115790 4424 tablet_bootstrap.cc:492] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Bootstrap starting.
I20251025 08:02:54.115833 4423 tablet_bootstrap.cc:492] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5: Bootstrap starting.
I20251025 08:02:54.116331 4422 tablet_bootstrap.cc:654] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:54.116578 4423 tablet_bootstrap.cc:654] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:54.116793 4424 tablet_bootstrap.cc:654] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:54.117640 4423 tablet_bootstrap.cc:492] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5: No bootstrap required, opened a new log
I20251025 08:02:54.117731 4423 ts_tablet_manager.cc:1403] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:54.117986 4424 tablet_bootstrap.cc:492] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: No bootstrap required, opened a new log
I20251025 08:02:54.118046 4424 ts_tablet_manager.cc:1403] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251025 08:02:54.118075 4423 raft_consensus.cc:359] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.118162 4423 raft_consensus.cc:385] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:54.118183 4423 raft_consensus.cc:740] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2826ef6b198d4fe79b769f1450049ec5, State: Initialized, Role: FOLLOWER
I20251025 08:02:54.118299 4423 consensus_queue.cc:260] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.118379 4424 raft_consensus.cc:359] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.118455 4424 raft_consensus.cc:385] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:54.118479 4424 raft_consensus.cc:740] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2e51eb82624cc4a3f5a8f51ceabfb7, State: Initialized, Role: FOLLOWER
I20251025 08:02:54.118566 4423 ts_tablet_manager.cc:1434] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 08:02:54.118515 4430 tablet_replica.cc:442] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } } }
I20251025 08:02:54.118600 4424 consensus_queue.cc:260] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.118660 4430 tablet_replica.cc:445] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:02:54.118881 4429 tablet_replica.cc:442] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } } }
I20251025 08:02:54.119021 4429 tablet_replica.cc:445] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:02:54.119395 4422 tablet_bootstrap.cc:492] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: No bootstrap required, opened a new log
I20251025 08:02:54.119508 4422 ts_tablet_manager.cc:1403] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251025 08:02:54.120024 4422 raft_consensus.cc:359] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.120143 4422 raft_consensus.cc:385] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:54.120189 4422 raft_consensus.cc:740] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abebaf13080d42eb9b7574bd956fb9ef, State: Initialized, Role: FOLLOWER
I20251025 08:02:54.120330 4422 consensus_queue.cc:260] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.120713 4422 ts_tablet_manager.cc:1434] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 08:02:54.120669 4428 tablet_replica.cc:442] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } } }
I20251025 08:02:54.120815 4428 tablet_replica.cc:445] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:02:54.118865 4424 ts_tablet_manager.cc:1434] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251025 08:02:54.146821 4428 raft_consensus.cc:493] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:02:54.146970 4428 raft_consensus.cc:515] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.148125 4428 leader_election.cc:290] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1:43221), 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113)
I20251025 08:02:54.151532 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.148813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:40113 (local address 127.27.132.2:46871)
1025 08:02:54.149051 (+ 238us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:54.149066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:54.149180 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:54.149430 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:54.149433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:54.149452 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:54.149689 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.149695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.150422 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.150426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:54.151039 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.151047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.151179 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.151204 (+ 25us) client_negotiation.cc:769] Sending connection context
1025 08:02:54.151258 (+ 54us) client_negotiation.cc:240] Negotiation successful
1025 08:02:54.151318 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20251025 08:02:54.151543 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.148370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:43221 (local address 127.27.132.2:39797)
1025 08:02:54.148785 (+ 415us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:54.148802 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:54.148919 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:54.149308 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:54.149310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:54.149328 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:54.149679 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.149690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.150454 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.150457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:54.151052 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.151059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.151190 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.151210 (+ 20us) client_negotiation.cc:769] Sending connection context
1025 08:02:54.151262 (+ 52us) client_negotiation.cc:240] Negotiation successful
1025 08:02:54.151321 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":110,"threads_started":1}
I20251025 08:02:54.151901 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.148931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:46871 (local address 127.27.132.3:40113)
1025 08:02:54.149052 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:54.149058 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:54.149124 (+ 66us) server_negotiation.cc:408] Connection header received
1025 08:02:54.149287 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:54.149291 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:54.149340 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:54.149410 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:54.149814 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.150294 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.151176 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.151677 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.151722 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:54.151770 (+ 48us) server_negotiation.cc:300] Negotiation successful
1025 08:02:54.151815 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251025 08:02:54.152042 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.148458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:39797 (local address 127.27.132.1:43221)
1025 08:02:54.148618 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:54.148623 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:54.148880 (+ 257us) server_negotiation.cc:408] Connection header received
1025 08:02:54.149077 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:54.149081 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:54.149130 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:54.149186 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:54.149839 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.150315 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.151176 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.151704 (+ 528us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.151762 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:54.151819 (+ 57us) server_negotiation.cc:300] Negotiation successful
1025 08:02:54.151893 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251025 08:02:54.152319 4348 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c7cf5a99e5548a39d5f142562536222" candidate_uuid: "abebaf13080d42eb9b7574bd956fb9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" is_pre_election: true
I20251025 08:02:54.152491 4348 raft_consensus.cc:2468] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abebaf13080d42eb9b7574bd956fb9ef in term 0.
I20251025 08:02:54.152536 4063 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c7cf5a99e5548a39d5f142562536222" candidate_uuid: "abebaf13080d42eb9b7574bd956fb9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2826ef6b198d4fe79b769f1450049ec5" is_pre_election: true
I20251025 08:02:54.152741 4063 raft_consensus.cc:2468] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abebaf13080d42eb9b7574bd956fb9ef in term 0.
I20251025 08:02:54.152877 4139 leader_election.cc:304] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b2e51eb82624cc4a3f5a8f51ceabfb7, abebaf13080d42eb9b7574bd956fb9ef; no voters:
I20251025 08:02:54.153151 4428 raft_consensus.cc:2804] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:02:54.153211 4428 raft_consensus.cc:493] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:02:54.153270 4428 raft_consensus.cc:3060] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:54.154508 4428 raft_consensus.cc:515] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.154975 4428 leader_election.cc:290] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [CANDIDATE]: Term 1 election: Requested vote from peers 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1:43221), 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113)
I20251025 08:02:54.155401 4063 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c7cf5a99e5548a39d5f142562536222" candidate_uuid: "abebaf13080d42eb9b7574bd956fb9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2826ef6b198d4fe79b769f1450049ec5"
I20251025 08:02:54.155508 4348 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c7cf5a99e5548a39d5f142562536222" candidate_uuid: "abebaf13080d42eb9b7574bd956fb9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7"
I20251025 08:02:54.155556 4063 raft_consensus.cc:3060] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:54.155642 4348 raft_consensus.cc:3060] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:54.156499 4348 raft_consensus.cc:2468] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abebaf13080d42eb9b7574bd956fb9ef in term 1.
I20251025 08:02:54.156538 4063 raft_consensus.cc:2468] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abebaf13080d42eb9b7574bd956fb9ef in term 1.
I20251025 08:02:54.156886 4139 leader_election.cc:304] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2826ef6b198d4fe79b769f1450049ec5, abebaf13080d42eb9b7574bd956fb9ef; no voters:
I20251025 08:02:54.157066 4428 raft_consensus.cc:2804] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:54.157271 4428 raft_consensus.cc:697] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [term 1 LEADER]: Becoming Leader. State: Replica: abebaf13080d42eb9b7574bd956fb9ef, State: Running, Role: LEADER
I20251025 08:02:54.157526 4428 consensus_queue.cc:237] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.158756 4464 tablet_replica.cc:442] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: TxnStatusTablet state changed. Reason: New leader abebaf13080d42eb9b7574bd956fb9ef. Latest consensus state: current_term: 1 leader_uuid: "abebaf13080d42eb9b7574bd956fb9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } } }
I20251025 08:02:54.158890 4464 tablet_replica.cc:445] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: This TxnStatusTablet replica's current role is: LEADER
I20251025 08:02:54.159169 4466 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251025 08:02:54.159302 4466 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251025 08:02:54.160069 3902 catalog_manager.cc:5649] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef reported cstate change: term changed from 0 to 1, leader changed from <none> to abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2). New cstate: current_term: 1 leader_uuid: "abebaf13080d42eb9b7574bd956fb9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20251025 08:02:54.178135 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.174934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.27.132.2:44491)
1025 08:02:54.175086 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:54.175091 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:54.175107 (+ 16us) server_negotiation.cc:408] Connection header received
1025 08:02:54.175277 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:54.175280 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:54.175377 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:54.175471 (+ 94us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:02:54.175839 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.176321 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.176991 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.177189 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.177241 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:02:54.177710 (+ 469us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:02:54.177820 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:54.177932 (+ 112us) server_negotiation.cc:300] Negotiation successful
1025 08:02:54.177987 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251025 08:02:54.178135 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.174799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:44491 (local address 127.0.0.1:37752)
1025 08:02:54.174968 (+ 169us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:54.174988 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:54.175123 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:54.175502 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:54.175505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:54.175513 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:02:54.175710 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.175716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.176443 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.176446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:54.176873 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.176880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.176991 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.177043 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:02:54.177824 (+ 781us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:02:54.177843 (+ 19us) client_negotiation.cc:769] Sending connection context
1025 08:02:54.177913 (+ 70us) client_negotiation.cc:240] Negotiation successful
1025 08:02:54.177999 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20251025 08:02:54.181275 4348 raft_consensus.cc:1275] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Refusing update from remote peer abebaf13080d42eb9b7574bd956fb9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:54.181442 4063 raft_consensus.cc:1275] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Refusing update from remote peer abebaf13080d42eb9b7574bd956fb9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:54.181911 4428 consensus_queue.cc:1048] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:54.182101 4464 consensus_queue.cc:1048] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:54.184103 4430 tablet_replica.cc:442] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5: TxnStatusTablet state changed. Reason: New leader abebaf13080d42eb9b7574bd956fb9ef. Latest consensus state: current_term: 1 leader_uuid: "abebaf13080d42eb9b7574bd956fb9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } } }
I20251025 08:02:54.184214 4430 tablet_replica.cc:445] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:02:54.184584 4429 tablet_replica.cc:442] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: TxnStatusTablet state changed. Reason: New leader abebaf13080d42eb9b7574bd956fb9ef. Latest consensus state: current_term: 1 leader_uuid: "abebaf13080d42eb9b7574bd956fb9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } } }
I20251025 08:02:54.184706 4429 tablet_replica.cc:445] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:02:54.188597 4464 tablet_replica.cc:442] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abebaf13080d42eb9b7574bd956fb9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } } }
I20251025 08:02:54.188755 4464 tablet_replica.cc:445] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: This TxnStatusTablet replica's current role is: LEADER
I20251025 08:02:54.189687 4428 tablet_replica.cc:442] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abebaf13080d42eb9b7574bd956fb9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } } }
I20251025 08:02:54.189807 4428 tablet_replica.cc:445] T 3c7cf5a99e5548a39d5f142562536222 P abebaf13080d42eb9b7574bd956fb9ef: This TxnStatusTablet replica's current role is: LEADER
I20251025 08:02:54.191056 4429 tablet_replica.cc:442] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abebaf13080d42eb9b7574bd956fb9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } } }
I20251025 08:02:54.191175 4429 tablet_replica.cc:445] T 3c7cf5a99e5548a39d5f142562536222 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:02:54.193369 4430 tablet_replica.cc:442] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abebaf13080d42eb9b7574bd956fb9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } } }
I20251025 08:02:54.193502 4430 tablet_replica.cc:445] T 3c7cf5a99e5548a39d5f142562536222 P 2826ef6b198d4fe79b769f1450049ec5: This TxnStatusTablet replica's current role is: FOLLOWER
I20251025 08:02:54.202296 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.199316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:40727 (local address 127.27.132.2:44491)
1025 08:02:54.199485 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:54.199496 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:54.199588 (+ 92us) server_negotiation.cc:408] Connection header received
1025 08:02:54.199719 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:54.199722 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:54.199769 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:54.199834 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:02:54.200287 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.200762 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.201655 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.201797 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.201854 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:02:54.201950 (+ 96us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:02:54.202012 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:54.202147 (+ 135us) server_negotiation.cc:300] Negotiation successful
1025 08:02:54.202190 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251025 08:02:54.202351 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.199170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:44491 (local address 127.27.132.1:40727)
1025 08:02:54.199501 (+ 331us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:54.199519 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:54.199633 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:54.199864 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:54.199867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:54.199876 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:02:54.200193 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.200201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.200883 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.200887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:54.201552 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.201561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.201674 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.201721 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:02:54.202055 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:02:54.202061 (+ 6us) client_negotiation.cc:769] Sending connection context
1025 08:02:54.202125 (+ 64us) client_negotiation.cc:240] Negotiation successful
1025 08:02:54.202199 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":105,"threads_started":1}
I20251025 08:02:54.209497 4348 raft_consensus.cc:1275] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Refusing update from remote peer 2826ef6b198d4fe79b769f1450049ec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:54.209540 4206 raft_consensus.cc:1275] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Refusing update from remote peer 2826ef6b198d4fe79b769f1450049ec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:54.209925 4430 consensus_queue.cc:1048] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:54.210083 4433 consensus_queue.cc:1048] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:54.232160 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.228894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:46065 (local address 127.27.132.1:43221)
1025 08:02:54.229027 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:54.229032 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:54.229213 (+ 181us) server_negotiation.cc:408] Connection header received
1025 08:02:54.229366 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:54.229371 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:54.229443 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:54.229520 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:02:54.230087 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.230732 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.231466 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.231665 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.231714 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:02:54.231792 (+ 78us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:02:54.231859 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:54.231969 (+ 110us) server_negotiation.cc:300] Negotiation successful
1025 08:02:54.232024 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251025 08:02:54.232393 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.228821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:43221 (local address 127.27.132.2:46065)
1025 08:02:54.229131 (+ 310us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:54.229144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:54.229259 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:54.229631 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:54.229634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:54.229643 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:02:54.229964 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.229971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.230869 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.230875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:54.231317 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.231363 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.231477 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.231537 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:02:54.231883 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:02:54.231889 (+ 6us) client_negotiation.cc:769] Sending connection context
1025 08:02:54.231954 (+ 65us) client_negotiation.cc:240] Negotiation successful
1025 08:02:54.232027 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
I20251025 08:02:54.256606 3902 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51274:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:54.256794 3902 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51274:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:54.258896 3902 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3: Sending DeleteTablet for 3 replicas of tablet 1ad7078ed28b49829d29038d5b7fbe1e
I20251025 08:02:54.259618 4328 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ad7078ed28b49829d29038d5b7fbe1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:54 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:41506
I20251025 08:02:54.259637 4186 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ad7078ed28b49829d29038d5b7fbe1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:54 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:37746
I20251025 08:02:54.259727 4040 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ad7078ed28b49829d29038d5b7fbe1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:54 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:33978
I20251025 08:02:54.259833 4447 tablet_replica.cc:333] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7: stopping tablet replica
I20251025 08:02:54.259922 4445 tablet_replica.cc:333] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef: stopping tablet replica
I20251025 08:02:54.259954 4447 raft_consensus.cc:2243] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:54.260039 4445 raft_consensus.cc:2243] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:54.260079 4447 raft_consensus.cc:2272] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:54.260154 4445 raft_consensus.cc:2272] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:54.260335 4446 tablet_replica.cc:333] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5: stopping tablet replica
I20251025 08:02:54.260449 4446 raft_consensus.cc:2243] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:02:54.260632 4446 raft_consensus.cc:2272] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:54.260798 4447 ts_tablet_manager.cc:1916] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:54.261344 4446 ts_tablet_manager.cc:1916] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:54.263581 4447 ts_tablet_manager.cc:1929] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:02:54.263649 4447 log.cc:1199] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/1ad7078ed28b49829d29038d5b7fbe1e
I20251025 08:02:54.263873 4445 ts_tablet_manager.cc:1916] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:54.263907 4447 ts_tablet_manager.cc:1950] T 1ad7078ed28b49829d29038d5b7fbe1e P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting consensus metadata
I20251025 08:02:54.264348 4446 ts_tablet_manager.cc:1929] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:02:54.264410 4446 log.cc:1199] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/wal/wals/1ad7078ed28b49829d29038d5b7fbe1e
I20251025 08:02:54.264675 4446 ts_tablet_manager.cc:1950] T 1ad7078ed28b49829d29038d5b7fbe1e P 2826ef6b198d4fe79b769f1450049ec5: Deleting consensus metadata
I20251025 08:02:54.264783 3891 catalog_manager.cc:4985] TS 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113): tablet 1ad7078ed28b49829d29038d5b7fbe1e (table test-table [id=da02e77b51c5455089836aa13fdcefe4]) successfully deleted
I20251025 08:02:54.265317 3891 catalog_manager.cc:4985] TS 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1:43221): tablet 1ad7078ed28b49829d29038d5b7fbe1e (table test-table [id=da02e77b51c5455089836aa13fdcefe4]) successfully deleted
I20251025 08:02:54.266628 4445 ts_tablet_manager.cc:1929] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:02:54.266700 4445 log.cc:1199] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/1ad7078ed28b49829d29038d5b7fbe1e
I20251025 08:02:54.266924 4445 ts_tablet_manager.cc:1950] T 1ad7078ed28b49829d29038d5b7fbe1e P abebaf13080d42eb9b7574bd956fb9ef: Deleting consensus metadata
I20251025 08:02:54.267575 3892 catalog_manager.cc:4985] TS abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2:44491): tablet 1ad7078ed28b49829d29038d5b7fbe1e (table test-table [id=da02e77b51c5455089836aa13fdcefe4]) successfully deleted
Oct 25 08:02:54 dist-test-slave-lq8l krb5kdc[3857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 25 08:02:54 dist-test-slave-lq8l krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:02:54 dist-test-slave-lq8l krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
W20251025 08:02:54.295578 4458 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:02:54.288591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.27.132.62:34111)
1025 08:02:54.288730 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:54.288735 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:54.289018 (+ 283us) server_negotiation.cc:408] Connection header received
1025 08:02:54.289134 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:54.289138 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:54.289199 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:54.289290 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:54.290287 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.291116 (+ 829us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.292043 (+ 927us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.292641 (+ 598us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.295432 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51300: BlockingRecv error: recv got EOF from 127.0.0.1:51300 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Oct 25 08:02:54 dist-test-slave-lq8l krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM
I20251025 08:02:54.304920 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.297162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.27.132.62:34111)
1025 08:02:54.297285 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:54.297290 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:54.297419 (+ 129us) server_negotiation.cc:408] Connection header received
1025 08:02:54.297566 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:54.297569 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:54.297611 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:54.297678 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:54.298353 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.298839 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.299438 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.299650 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.301970 (+ 2320us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:54.301988 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:54.301991 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:54.302029 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:54.303523 (+ 1494us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:54.303894 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:54.303897 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:54.303898 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:54.303939 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:54.304140 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:54.304143 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:54.304145 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:54.304291 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:54.304412 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:54.304628 (+ 216us) server_negotiation.cc:300] Negotiation successful
1025 08:02:54.304750 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 08:02:54.307575 3902 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:02:54.308113 3902 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:02:54.313021 4040 tablet_service.cc:1505] Processing CreateTablet for tablet f1bc498ec5f64a73aaf3256f981d2781 (DEFAULT_TABLE table=test-table [id=647afe33a3754feda17395761071620d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:54.313072 4186 tablet_service.cc:1505] Processing CreateTablet for tablet f1bc498ec5f64a73aaf3256f981d2781 (DEFAULT_TABLE table=test-table [id=647afe33a3754feda17395761071620d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:54.313345 4186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1bc498ec5f64a73aaf3256f981d2781. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:54.313354 4040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1bc498ec5f64a73aaf3256f981d2781. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:54.315114 4422 tablet_bootstrap.cc:492] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef: Bootstrap starting.
I20251025 08:02:54.315760 4423 tablet_bootstrap.cc:492] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5: Bootstrap starting.
I20251025 08:02:54.316110 4422 tablet_bootstrap.cc:654] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:54.316705 4423 tablet_bootstrap.cc:654] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:54.317369 4328 tablet_service.cc:1505] Processing CreateTablet for tablet f1bc498ec5f64a73aaf3256f981d2781 (DEFAULT_TABLE table=test-table [id=647afe33a3754feda17395761071620d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:54.317551 4422 tablet_bootstrap.cc:492] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef: No bootstrap required, opened a new log
I20251025 08:02:54.317651 4328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1bc498ec5f64a73aaf3256f981d2781. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:54.317647 4422 ts_tablet_manager.cc:1403] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251025 08:02:54.318043 4422 raft_consensus.cc:359] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.318140 4422 raft_consensus.cc:385] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:54.318176 4422 raft_consensus.cc:740] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abebaf13080d42eb9b7574bd956fb9ef, State: Initialized, Role: FOLLOWER
I20251025 08:02:54.318300 4422 consensus_queue.cc:260] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.318532 4422 ts_tablet_manager.cc:1434] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:02:54.320014 4424 tablet_bootstrap.cc:492] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Bootstrap starting.
I20251025 08:02:54.320477 4423 tablet_bootstrap.cc:492] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5: No bootstrap required, opened a new log
I20251025 08:02:54.320564 4423 ts_tablet_manager.cc:1403] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20251025 08:02:54.321058 4424 tablet_bootstrap.cc:654] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:54.321036 4423 raft_consensus.cc:359] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.321141 4423 raft_consensus.cc:385] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:54.321175 4423 raft_consensus.cc:740] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2826ef6b198d4fe79b769f1450049ec5, State: Initialized, Role: FOLLOWER
I20251025 08:02:54.321308 4423 consensus_queue.cc:260] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.321624 4423 ts_tablet_manager.cc:1434] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 08:02:54.323546 4424 tablet_bootstrap.cc:492] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: No bootstrap required, opened a new log
I20251025 08:02:54.323647 4424 ts_tablet_manager.cc:1403] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20251025 08:02:54.324043 4424 raft_consensus.cc:359] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.324136 4424 raft_consensus.cc:385] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:54.324170 4424 raft_consensus.cc:740] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2e51eb82624cc4a3f5a8f51ceabfb7, State: Initialized, Role: FOLLOWER
I20251025 08:02:54.324337 4424 consensus_queue.cc:260] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.324604 4424 ts_tablet_manager.cc:1434] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 08:02:54.639052 4429 raft_consensus.cc:493] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:02:54.639195 4429 raft_consensus.cc:515] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.640326 4429 leader_election.cc:290] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1:43221), abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2:44491)
I20251025 08:02:54.644122 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.640554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:43221 (local address 127.27.132.3:55959)
1025 08:02:54.640831 (+ 277us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:54.640847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:54.640977 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:54.641260 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:54.641264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:54.641287 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:54.641636 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.641644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.642624 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.642627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:54.643647 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.643657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.643784 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.643808 (+ 24us) client_negotiation.cc:769] Sending connection context
1025 08:02:54.643863 (+ 55us) client_negotiation.cc:240] Negotiation successful
1025 08:02:54.643932 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"spinlock_wait_cycles":23296,"thread_start_us":74,"threads_started":1}
I20251025 08:02:54.644135 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.640589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:44491 (local address 127.27.132.3:53817)
1025 08:02:54.640847 (+ 258us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:54.640860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:54.640976 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:54.641260 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:54.641263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:54.641289 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:02:54.641636 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.641644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.642624 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.642627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:54.643646 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:54.643657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.643785 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.643808 (+ 23us) client_negotiation.cc:769] Sending connection context
1025 08:02:54.643863 (+ 55us) client_negotiation.cc:240] Negotiation successful
1025 08:02:54.643932 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"spinlock_wait_cycles":29312,"thread_start_us":76,"threads_started":1}
I20251025 08:02:54.644543 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.640668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:53817 (local address 127.27.132.2:44491)
1025 08:02:54.640883 (+ 215us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:54.640888 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:54.640902 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:02:54.641089 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:54.641093 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:54.641148 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:54.641234 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:54.641749 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.642516 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.643790 (+ 1274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.644277 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.644311 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:54.644366 (+ 55us) server_negotiation.cc:300] Negotiation successful
1025 08:02:54.644431 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20251025 08:02:54.644536 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.640648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:55959 (local address 127.27.132.1:43221)
1025 08:02:54.640844 (+ 196us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:54.640849 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:54.640932 (+ 83us) server_negotiation.cc:408] Connection header received
1025 08:02:54.641089 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:54.641093 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:54.641148 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:54.641234 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:02:54.641748 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.642509 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.643790 (+ 1281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.644272 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.644311 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:54.644365 (+ 54us) server_negotiation.cc:300] Negotiation successful
1025 08:02:54.644421 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20251025 08:02:54.644981 4206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1bc498ec5f64a73aaf3256f981d2781" candidate_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abebaf13080d42eb9b7574bd956fb9ef" is_pre_election: true
I20251025 08:02:54.644999 4063 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1bc498ec5f64a73aaf3256f981d2781" candidate_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2826ef6b198d4fe79b769f1450049ec5" is_pre_election: true
I20251025 08:02:54.645125 4206 raft_consensus.cc:2468] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2e51eb82624cc4a3f5a8f51ceabfb7 in term 0.
I20251025 08:02:54.645138 4063 raft_consensus.cc:2468] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2e51eb82624cc4a3f5a8f51ceabfb7 in term 0.
I20251025 08:02:54.645502 4281 leader_election.cc:304] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2826ef6b198d4fe79b769f1450049ec5, 6b2e51eb82624cc4a3f5a8f51ceabfb7; no voters:
I20251025 08:02:54.645736 4429 raft_consensus.cc:2804] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:02:54.645798 4429 raft_consensus.cc:493] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:02:54.645835 4429 raft_consensus.cc:3060] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:54.646726 4429 raft_consensus.cc:515] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.647074 4429 leader_election.cc:290] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [CANDIDATE]: Term 1 election: Requested vote from peers 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1:43221), abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2:44491)
I20251025 08:02:54.647444 4206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1bc498ec5f64a73aaf3256f981d2781" candidate_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abebaf13080d42eb9b7574bd956fb9ef"
I20251025 08:02:54.647444 4063 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1bc498ec5f64a73aaf3256f981d2781" candidate_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2826ef6b198d4fe79b769f1450049ec5"
I20251025 08:02:54.647571 4063 raft_consensus.cc:3060] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:54.647572 4206 raft_consensus.cc:3060] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:54.648633 4063 raft_consensus.cc:2468] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2e51eb82624cc4a3f5a8f51ceabfb7 in term 1.
I20251025 08:02:54.648633 4206 raft_consensus.cc:2468] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2e51eb82624cc4a3f5a8f51ceabfb7 in term 1.
I20251025 08:02:54.648924 4281 leader_election.cc:304] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2826ef6b198d4fe79b769f1450049ec5, 6b2e51eb82624cc4a3f5a8f51ceabfb7; no voters:
I20251025 08:02:54.649102 4429 raft_consensus.cc:2804] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:54.649276 4429 raft_consensus.cc:697] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 LEADER]: Becoming Leader. State: Replica: 6b2e51eb82624cc4a3f5a8f51ceabfb7, State: Running, Role: LEADER
I20251025 08:02:54.649514 4429 consensus_queue.cc:237] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.651878 3905 catalog_manager.cc:5649] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3). New cstate: current_term: 1 leader_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } health_report { overall_health: HEALTHY } } }
I20251025 08:02:54.690383 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:54.687524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.27.132.3:40113)
1025 08:02:54.687751 (+ 227us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:54.687755 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:54.687775 (+ 20us) server_negotiation.cc:408] Connection header received
1025 08:02:54.687857 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:54.687860 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:54.687917 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:54.688009 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:02:54.688285 (+ 276us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.688790 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:54.689347 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:54.689477 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:54.689526 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:02:54.689998 (+ 472us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:02:54.690118 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:54.690148 (+ 30us) server_negotiation.cc:300] Negotiation successful
1025 08:02:54.690199 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20251025 08:02:54.692885 4206 raft_consensus.cc:1275] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Refusing update from remote peer 6b2e51eb82624cc4a3f5a8f51ceabfb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:54.693025 4063 raft_consensus.cc:1275] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Refusing update from remote peer 6b2e51eb82624cc4a3f5a8f51ceabfb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:54.693357 4498 consensus_queue.cc:1048] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:54.693559 4429 consensus_queue.cc:1048] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:54.696295 4438 mvcc.cc:204] Tried to move back new op lower bound from 7214609918737477632 to 7214609918566629376. Current Snapshot: MvccSnapshot[applied={T|T < 7214609918737477632}]
I20251025 08:02:54.703451 3902 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51316:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:54.703621 3902 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51316:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:54.705614 3902 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3: Sending DeleteTablet for 3 replicas of tablet f1bc498ec5f64a73aaf3256f981d2781
I20251025 08:02:54.706171 4328 tablet_service.cc:1552] Processing DeleteTablet for tablet f1bc498ec5f64a73aaf3256f981d2781 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:54 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:41506
I20251025 08:02:54.706259 4040 tablet_service.cc:1552] Processing DeleteTablet for tablet f1bc498ec5f64a73aaf3256f981d2781 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:54 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:33978
I20251025 08:02:54.706359 4186 tablet_service.cc:1552] Processing DeleteTablet for tablet f1bc498ec5f64a73aaf3256f981d2781 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:54 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:37746
I20251025 08:02:54.706455 4446 tablet_replica.cc:333] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5: stopping tablet replica
I20251025 08:02:54.706553 4447 tablet_replica.cc:333] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: stopping tablet replica
I20251025 08:02:54.706585 4446 raft_consensus.cc:2243] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:54.706668 4447 raft_consensus.cc:2243] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:02:54.706701 4446 raft_consensus.cc:2272] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:54.706967 4447 raft_consensus.cc:2272] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:54.707166 4445 tablet_replica.cc:333] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef: stopping tablet replica
I20251025 08:02:54.707269 4445 raft_consensus.cc:2243] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:54.707422 4445 raft_consensus.cc:2272] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:54.707418 4446 ts_tablet_manager.cc:1916] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:54.707700 4447 ts_tablet_manager.cc:1916] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:54.708297 3902 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:02:54.708712 3902 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:02:54.709769 4446 ts_tablet_manager.cc:1929] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:54.709769 4447 ts_tablet_manager.cc:1929] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:54.709837 4447 log.cc:1199] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/f1bc498ec5f64a73aaf3256f981d2781
I20251025 08:02:54.709837 4446 log.cc:1199] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/wal/wals/f1bc498ec5f64a73aaf3256f981d2781
I20251025 08:02:54.710090 4446 ts_tablet_manager.cc:1950] T f1bc498ec5f64a73aaf3256f981d2781 P 2826ef6b198d4fe79b769f1450049ec5: Deleting consensus metadata
I20251025 08:02:54.710090 4447 ts_tablet_manager.cc:1950] T f1bc498ec5f64a73aaf3256f981d2781 P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting consensus metadata
I20251025 08:02:54.710350 4445 ts_tablet_manager.cc:1916] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:54.710914 3891 catalog_manager.cc:4985] TS 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1:43221): tablet f1bc498ec5f64a73aaf3256f981d2781 (table test-table [id=647afe33a3754feda17395761071620d]) successfully deleted
I20251025 08:02:54.712658 4445 ts_tablet_manager.cc:1929] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:02:54.712733 4445 log.cc:1199] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/f1bc498ec5f64a73aaf3256f981d2781
I20251025 08:02:54.712860 3891 catalog_manager.cc:4985] TS 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113): tablet f1bc498ec5f64a73aaf3256f981d2781 (table test-table [id=647afe33a3754feda17395761071620d]) successfully deleted
I20251025 08:02:54.712989 4445 ts_tablet_manager.cc:1950] T f1bc498ec5f64a73aaf3256f981d2781 P abebaf13080d42eb9b7574bd956fb9ef: Deleting consensus metadata
I20251025 08:02:54.713645 3892 catalog_manager.cc:4985] TS abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2:44491): tablet f1bc498ec5f64a73aaf3256f981d2781 (table test-table [id=647afe33a3754feda17395761071620d]) successfully deleted
I20251025 08:02:54.715626 4040 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1ef0fec72c4a6ba941c091721f20fc (DEFAULT_TABLE table=test-table [id=058eba7858124cfcaf2cfd5bd942ae5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:54.715919 4040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1ef0fec72c4a6ba941c091721f20fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:54.715895 4186 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1ef0fec72c4a6ba941c091721f20fc (DEFAULT_TABLE table=test-table [id=058eba7858124cfcaf2cfd5bd942ae5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:54.716122 4186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1ef0fec72c4a6ba941c091721f20fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:54.717720 4422 tablet_bootstrap.cc:492] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef: Bootstrap starting.
I20251025 08:02:54.718164 4423 tablet_bootstrap.cc:492] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5: Bootstrap starting.
I20251025 08:02:54.718638 4422 tablet_bootstrap.cc:654] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:54.719048 4423 tablet_bootstrap.cc:654] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:54.719185 4328 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1ef0fec72c4a6ba941c091721f20fc (DEFAULT_TABLE table=test-table [id=058eba7858124cfcaf2cfd5bd942ae5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:02:54.719497 4328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1ef0fec72c4a6ba941c091721f20fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:54.720144 4423 tablet_bootstrap.cc:492] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5: No bootstrap required, opened a new log
I20251025 08:02:54.720238 4423 ts_tablet_manager.cc:1403] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:54.720616 4423 raft_consensus.cc:359] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.720716 4423 raft_consensus.cc:385] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:54.720777 4423 raft_consensus.cc:740] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2826ef6b198d4fe79b769f1450049ec5, State: Initialized, Role: FOLLOWER
I20251025 08:02:54.720914 4423 consensus_queue.cc:260] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.721172 4423 ts_tablet_manager.cc:1434] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:02:54.721805 4424 tablet_bootstrap.cc:492] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Bootstrap starting.
I20251025 08:02:54.722743 4424 tablet_bootstrap.cc:654] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:54.724330 4422 tablet_bootstrap.cc:492] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef: No bootstrap required, opened a new log
I20251025 08:02:54.724462 4422 ts_tablet_manager.cc:1403] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251025 08:02:54.724855 4422 raft_consensus.cc:359] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.724952 4422 raft_consensus.cc:385] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:54.725013 4422 raft_consensus.cc:740] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abebaf13080d42eb9b7574bd956fb9ef, State: Initialized, Role: FOLLOWER
I20251025 08:02:54.725083 4424 tablet_bootstrap.cc:492] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7: No bootstrap required, opened a new log
I20251025 08:02:54.725167 4424 ts_tablet_manager.cc:1403] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251025 08:02:54.725178 4422 consensus_queue.cc:260] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.725486 4422 ts_tablet_manager.cc:1434] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:02:54.725630 4424 raft_consensus.cc:359] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.725744 4424 raft_consensus.cc:385] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:54.725776 4424 raft_consensus.cc:740] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2e51eb82624cc4a3f5a8f51ceabfb7, State: Initialized, Role: FOLLOWER
I20251025 08:02:54.725893 4424 consensus_queue.cc:260] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:54.726117 4424 ts_tablet_manager.cc:1434] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 08:02:55.061960 4464 raft_consensus.cc:493] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:02:55.062140 4464 raft_consensus.cc:515] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:55.062597 4464 leader_election.cc:290] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1:43221), 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113)
I20251025 08:02:55.063112 4063 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1ef0fec72c4a6ba941c091721f20fc" candidate_uuid: "abebaf13080d42eb9b7574bd956fb9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2826ef6b198d4fe79b769f1450049ec5" is_pre_election: true
I20251025 08:02:55.063174 4348 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1ef0fec72c4a6ba941c091721f20fc" candidate_uuid: "abebaf13080d42eb9b7574bd956fb9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" is_pre_election: true
I20251025 08:02:55.063241 4063 raft_consensus.cc:2468] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abebaf13080d42eb9b7574bd956fb9ef in term 0.
I20251025 08:02:55.063369 4348 raft_consensus.cc:2468] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abebaf13080d42eb9b7574bd956fb9ef in term 0.
I20251025 08:02:55.063585 4139 leader_election.cc:304] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2826ef6b198d4fe79b769f1450049ec5, abebaf13080d42eb9b7574bd956fb9ef; no voters:
I20251025 08:02:55.063791 4464 raft_consensus.cc:2804] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:02:55.063848 4464 raft_consensus.cc:493] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:02:55.063903 4464 raft_consensus.cc:3060] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:55.064822 4464 raft_consensus.cc:515] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:55.065140 4464 leader_election.cc:290] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [CANDIDATE]: Term 1 election: Requested vote from peers 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1:43221), 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113)
I20251025 08:02:55.065554 4063 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1ef0fec72c4a6ba941c091721f20fc" candidate_uuid: "abebaf13080d42eb9b7574bd956fb9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2826ef6b198d4fe79b769f1450049ec5"
I20251025 08:02:55.065579 4348 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1ef0fec72c4a6ba941c091721f20fc" candidate_uuid: "abebaf13080d42eb9b7574bd956fb9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7"
I20251025 08:02:55.065666 4063 raft_consensus.cc:3060] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:55.065680 4348 raft_consensus.cc:3060] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:55.066514 4348 raft_consensus.cc:2468] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abebaf13080d42eb9b7574bd956fb9ef in term 1.
I20251025 08:02:55.066519 4063 raft_consensus.cc:2468] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abebaf13080d42eb9b7574bd956fb9ef in term 1.
I20251025 08:02:55.066828 4139 leader_election.cc:304] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2826ef6b198d4fe79b769f1450049ec5, abebaf13080d42eb9b7574bd956fb9ef; no voters:
I20251025 08:02:55.066993 4464 raft_consensus.cc:2804] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:55.067170 4464 raft_consensus.cc:697] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 1 LEADER]: Becoming Leader. State: Replica: abebaf13080d42eb9b7574bd956fb9ef, State: Running, Role: LEADER
I20251025 08:02:55.067314 4464 consensus_queue.cc:237] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } }
I20251025 08:02:55.068974 3902 catalog_manager.cc:5649] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef reported cstate change: term changed from 0 to 1, leader changed from <none> to abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2). New cstate: current_term: 1 leader_uuid: "abebaf13080d42eb9b7574bd956fb9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abebaf13080d42eb9b7574bd956fb9ef" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 44491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20251025 08:02:55.095667 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:55.093211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.27.132.2:44491)
1025 08:02:55.093333 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:55.093337 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:55.093353 (+ 16us) server_negotiation.cc:408] Connection header received
1025 08:02:55.093474 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:55.093477 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:55.093524 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:55.093589 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:02:55.093966 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:55.094428 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:55.095062 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:55.095185 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:55.095222 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:02:55.095286 (+ 64us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:02:55.095353 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:55.095469 (+ 116us) server_negotiation.cc:300] Negotiation successful
1025 08:02:55.095511 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 08:02:55.104748 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:55.102037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:40735 (local address 127.27.132.2:44491)
1025 08:02:55.102252 (+ 215us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:55.102258 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:55.102456 (+ 198us) server_negotiation.cc:408] Connection header received
1025 08:02:55.102565 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:55.102567 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:55.102611 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:55.102675 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:02:55.102980 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:55.103472 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:55.104150 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:55.104286 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:55.104322 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:02:55.104383 (+ 61us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:02:55.104441 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:55.104603 (+ 162us) server_negotiation.cc:300] Negotiation successful
1025 08:02:55.104642 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20251025 08:02:55.104753 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:55.101974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:44491 (local address 127.27.132.2:40735)
1025 08:02:55.102363 (+ 389us) negotiation.cc:107] Waiting for socket to connect
1025 08:02:55.102376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:02:55.102490 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:02:55.102692 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:02:55.102694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:02:55.102701 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1025 08:02:55.102899 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:55.102903 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:55.103573 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:02:55.103575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:02:55.104048 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:02:55.104055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:02:55.104142 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:55.104165 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1025 08:02:55.104493 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1025 08:02:55.104499 (+ 6us) client_negotiation.cc:769] Sending connection context
1025 08:02:55.104576 (+ 77us) client_negotiation.cc:240] Negotiation successful
1025 08:02:55.104632 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":120,"threads_started":1}
I20251025 08:02:55.110678 4348 raft_consensus.cc:1275] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Refusing update from remote peer abebaf13080d42eb9b7574bd956fb9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:55.110718 4063 raft_consensus.cc:1275] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Refusing update from remote peer abebaf13080d42eb9b7574bd956fb9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:02:55.111104 4509 consensus_queue.cc:1048] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2e51eb82624cc4a3f5a8f51ceabfb7" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:55.111259 4511 consensus_queue.cc:1048] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2826ef6b198d4fe79b769f1450049ec5" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 43221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:02:55.146365 3903 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51316:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:55.146538 3903 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51316:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:02:55.148795 3903 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 46c0e57e26994ed8909a1c92d2a7c5a3: Sending DeleteTablet for 3 replicas of tablet 2a1ef0fec72c4a6ba941c091721f20fc
I20251025 08:02:55.149434 4040 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a1ef0fec72c4a6ba941c091721f20fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:55 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:33978
I20251025 08:02:55.149564 4328 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a1ef0fec72c4a6ba941c091721f20fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:55 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:41506
I20251025 08:02:55.149627 4185 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a1ef0fec72c4a6ba941c091721f20fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:02:55 UTC) from {username='oryx', principal='oryx/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:37746
I20251025 08:02:55.149655 4446 tablet_replica.cc:333] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5: stopping tablet replica
I20251025 08:02:55.149777 4446 raft_consensus.cc:2243] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:55.149791 4445 tablet_replica.cc:333] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef: stopping tablet replica
I20251025 08:02:55.149791 4447 tablet_replica.cc:333] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7: stopping tablet replica
I20251025 08:02:55.149893 4446 raft_consensus.cc:2272] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:55.149894 4445 raft_consensus.cc:2243] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:02:55.149894 4447 raft_consensus.cc:2243] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:02:55.150009 4447 raft_consensus.cc:2272] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:55.150229 4445 raft_consensus.cc:2272] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:02:55.151044 4447 ts_tablet_manager.cc:1916] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:55.151044 4445 ts_tablet_manager.cc:1916] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:55.151038 4446 ts_tablet_manager.cc:1916] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:02:55.153481 4445 ts_tablet_manager.cc:1929] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:02:55.153481 4447 ts_tablet_manager.cc:1929] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:02:55.153481 4446 ts_tablet_manager.cc:1929] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251025 08:02:55.153568 4447 log.cc:1199] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/2a1ef0fec72c4a6ba941c091721f20fc
I20251025 08:02:55.153568 4445 log.cc:1199] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/2a1ef0fec72c4a6ba941c091721f20fc
I20251025 08:02:55.153573 4446 log.cc:1199] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761379300881738-28176-0/minicluster-data/ts-0/wal/wals/2a1ef0fec72c4a6ba941c091721f20fc
I20251025 08:02:55.153844 4445 ts_tablet_manager.cc:1950] T 2a1ef0fec72c4a6ba941c091721f20fc P abebaf13080d42eb9b7574bd956fb9ef: Deleting consensus metadata
I20251025 08:02:55.153844 4447 ts_tablet_manager.cc:1950] T 2a1ef0fec72c4a6ba941c091721f20fc P 6b2e51eb82624cc4a3f5a8f51ceabfb7: Deleting consensus metadata
I20251025 08:02:55.153846 4446 ts_tablet_manager.cc:1950] T 2a1ef0fec72c4a6ba941c091721f20fc P 2826ef6b198d4fe79b769f1450049ec5: Deleting consensus metadata
I20251025 08:02:55.154613 3892 catalog_manager.cc:4985] TS abebaf13080d42eb9b7574bd956fb9ef (127.27.132.2:44491): tablet 2a1ef0fec72c4a6ba941c091721f20fc (table test-table [id=058eba7858124cfcaf2cfd5bd942ae5b]) successfully deleted
I20251025 08:02:55.154629 3891 catalog_manager.cc:4985] TS 2826ef6b198d4fe79b769f1450049ec5 (127.27.132.1:43221): tablet 2a1ef0fec72c4a6ba941c091721f20fc (table test-table [id=058eba7858124cfcaf2cfd5bd942ae5b]) successfully deleted
I20251025 08:02:55.155197 3891 catalog_manager.cc:4985] TS 6b2e51eb82624cc4a3f5a8f51ceabfb7 (127.27.132.3:40113): tablet 2a1ef0fec72c4a6ba941c091721f20fc (table test-table [id=058eba7858124cfcaf2cfd5bd942ae5b]) successfully deleted
I20251025 08:02:55.155712 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 3979
I20251025 08:02:55.165555 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 4121
I20251025 08:02:55.171748 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 4263
I20251025 08:02:55.178223 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 3873
2025-10-25T08:02:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4714 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:02:55 dist-test-slave-lq8l krb5kdc[4522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:02:55 dist-test-slave-lq8l krb5kdc[4522](info): set up 2 sockets
Oct 25 08:02:55 dist-test-slave-lq8l krb5kdc[4522](info): commencing operation
krb5kdc: starting...
W20251025 08:02:57.160730 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.960s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:02:57 dist-test-slave-lq8l krb5kdc[4522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:02:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:57.291479 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:35165
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:35165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:57.396178 4538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:57.396430 4538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:57.396528 4538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:57.399837 4538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:57.399911 4538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:57.399950 4538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:57.399991 4538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:57.400028 4538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:57.404270 4538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:35165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:35165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:57.405398 4538 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:57.406271 4538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:57.412051 4543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:57.412061 4546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:57.412123 4538 server_base.cc:1047] running on GCE node
W20251025 08:02:57.412061 4544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:57.412812 4538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:57.413730 4538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:57.414887 4538 hybrid_clock.cc:648] HybridClock initialized: now 1761379377414871 us; error 45 us; skew 500 ppm
W20251025 08:02:57.417989 4538 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-10-25T08:02:57Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2239 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:02:57 dist-test-slave-lq8l krb5kdc[4551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:02:57 dist-test-slave-lq8l krb5kdc[4551](info): set up 2 sockets
Oct 25 08:02:57 dist-test-slave-lq8l krb5kdc[4551](info): commencing operation
krb5kdc: starting...
W20251025 08:02:59.416750 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.976s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:02:59 dist-test-slave-lq8l krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:02:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:02:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:59.552300 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:46479
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:46479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:59.656342 4567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:59.656567 4567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:59.656663 4567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:59.659828 4567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:02:59.659893 4567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:59.659929 4567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:59.659962 4567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:02:59.660001 4567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:02:59.664070 4567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:46479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:59.665123 4567 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:59.665943 4567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:59.671352 4572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:59.671350 4573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:59.671607 4567 server_base.cc:1047] running on GCE node
W20251025 08:02:59.671345 4575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:59.672184 4567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:59.673031 4567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:59.674170 4567 hybrid_clock.cc:648] HybridClock initialized: now 1761379379674156 us; error 37 us; skew 500 ppm
Oct 25 08:02:59 dist-test-slave-lq8l krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379379, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:59.676635 4567 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:02:59.677666 4567 webserver.cc:492] Webserver started at http://127.27.132.62:38231/ using document root <none> and password file <none>
I20251025 08:02:59.678195 4567 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:59.678249 4567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:59.678428 4567 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:59.680109 4567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "c834ec353ab64e6f9769f445da9665a6"
format_stamp: "Formatted at 2025-10-25 08:02:59 on dist-test-slave-lq8l"
server_key: "5764bb17f448a933ed88d3e6d3c74589"
server_key_iv: "02490f4689c8e4fc6540ab9afa993fb0"
server_key_version: "encryptionkey@0"
I20251025 08:02:59.680570 4567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "c834ec353ab64e6f9769f445da9665a6"
format_stamp: "Formatted at 2025-10-25 08:02:59 on dist-test-slave-lq8l"
server_key: "5764bb17f448a933ed88d3e6d3c74589"
server_key_iv: "02490f4689c8e4fc6540ab9afa993fb0"
server_key_version: "encryptionkey@0"
I20251025 08:02:59.683935 4567 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:02:59.686235 4582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:59.687386 4567 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251025 08:02:59.687516 4567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "c834ec353ab64e6f9769f445da9665a6"
format_stamp: "Formatted at 2025-10-25 08:02:59 on dist-test-slave-lq8l"
server_key: "5764bb17f448a933ed88d3e6d3c74589"
server_key_iv: "02490f4689c8e4fc6540ab9afa993fb0"
server_key_version: "encryptionkey@0"
I20251025 08:02:59.687613 4567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:02:59.714453 4567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:02:59.721004 4567 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:02:59.721201 4567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:02:59.728827 4567 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:46479
I20251025 08:02:59.728863 4634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:46479 every 8 connection(s)
I20251025 08:02:59.729859 4567 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:02:59.732666 4635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:02:59.737993 4635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6: Bootstrap starting.
I20251025 08:02:59.738287 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 4567
I20251025 08:02:59.738415 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:02:59.738651 28176 external_mini_cluster.cc:1468] Setting key 7d4e913dde628319c7a2f9ccf9ed6fa3
I20251025 08:02:59.740422 4635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6: Neither blocks nor log segments found. Creating new log.
I20251025 08:02:59.741261 4635 log.cc:826] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6: Log is configured to *not* fsync() on all Append() calls
I20251025 08:02:59.742980 4635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6: No bootstrap required, opened a new log
I20251025 08:02:59.745622 4635 raft_consensus.cc:359] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } }
I20251025 08:02:59.745818 4635 raft_consensus.cc:385] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:02:59.745889 4635 raft_consensus.cc:740] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c834ec353ab64e6f9769f445da9665a6, State: Initialized, Role: FOLLOWER
Oct 25 08:02:59 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:02:59.746328 4635 consensus_queue.cc:260] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } }
I20251025 08:02:59.746483 4635 raft_consensus.cc:399] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:02:59.746559 4635 raft_consensus.cc:493] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:02:59.746654 4635 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:02:59.747762 4635 raft_consensus.cc:515] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } }
I20251025 08:02:59.748168 4635 leader_election.cc:304] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c834ec353ab64e6f9769f445da9665a6; no voters:
I20251025 08:02:59.748538 4635 leader_election.cc:290] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:02:59.748631 4640 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:02:59.748844 4640 raft_consensus.cc:697] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 1 LEADER]: Becoming Leader. State: Replica: c834ec353ab64e6f9769f445da9665a6, State: Running, Role: LEADER
I20251025 08:02:59.749137 4640 consensus_queue.cc:237] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } }
I20251025 08:02:59.749761 4635 sys_catalog.cc:565] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:02:59.750756 4642 sys_catalog.cc:455] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c834ec353ab64e6f9769f445da9665a6. Latest consensus state: current_term: 1 leader_uuid: "c834ec353ab64e6f9769f445da9665a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } } }
I20251025 08:02:59.750873 4642 sys_catalog.cc:458] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:59.751111 4641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c834ec353ab64e6f9769f445da9665a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } } }
I20251025 08:02:59.751196 4641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [sys.catalog]: This master's current role is: LEADER
I20251025 08:02:59.752946 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:02:59.739897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.27.132.62:46479)
1025 08:02:59.740337 (+ 440us) server_negotiation.cc:207] Beginning negotiation
1025 08:02:59.740350 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1025 08:02:59.740396 (+ 46us) server_negotiation.cc:408] Connection header received
1025 08:02:59.741098 (+ 702us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:02:59.741123 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:02:59.741481 (+ 358us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:02:59.741844 (+ 363us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:02:59.742759 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:59.743586 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:02:59.744231 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:02:59.744528 (+ 297us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:02:59.746726 (+ 2198us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:02:59.746758 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:02:59.746771 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:02:59.746815 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:02:59.749531 (+ 2716us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:59.750033 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:59.750039 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:59.750044 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:59.750146 (+ 102us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:02:59.750461 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:02:59.750464 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:02:59.750466 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:02:59.750954 (+ 488us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:02:59.751179 (+ 225us) server_negotiation.cc:1036] Waiting for connection context
1025 08:02:59.751502 (+ 323us) server_negotiation.cc:300] Negotiation successful
1025 08:02:59.751969 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":123,"threads_started":1}
W20251025 08:02:59.754755 4655 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 08:02:59.754832 4655 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 08:02:59.754899 4649 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:02:59.755826 4649 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:02:59.761116 4649 catalog_manager.cc:1357] Generated new cluster ID: ec43db44a2144dda81442a316bd8f163
I20251025 08:02:59.761183 4649 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:02:59.784709 4649 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:02:59.785691 4649 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:02:59.793735 4649 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6: Generated new TSK 0
I20251025 08:02:59.794466 4649 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:02:59.842252 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:46479
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:02:59.950258 4663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:02:59.950510 4663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:02:59.950584 4663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:02:59.953868 4663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:02:59.953938 4663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:02:59.954048 4663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:02:59.958496 4663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:02:59.959591 4663 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:02:59.960434 4663 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:02:59.966683 4669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:59.966681 4671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:02:59.966678 4668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:02:59.966991 4663 server_base.cc:1047] running on GCE node
I20251025 08:02:59.967301 4663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:02:59.967834 4663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:02:59.968976 4663 hybrid_clock.cc:648] HybridClock initialized: now 1761379379968952 us; error 38 us; skew 500 ppm
Oct 25 08:02:59 dist-test-slave-lq8l krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379379, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:02:59.971482 4663 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:02:59.972460 4663 webserver.cc:492] Webserver started at http://127.27.132.1:42801/ using document root <none> and password file <none>
I20251025 08:02:59.972985 4663 fs_manager.cc:362] Metadata directory not provided
I20251025 08:02:59.973038 4663 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:02:59.973220 4663 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:02:59.974884 4663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "2bb6c9ea3e8c44fbae8e9cf7da26c365"
format_stamp: "Formatted at 2025-10-25 08:02:59 on dist-test-slave-lq8l"
server_key: "6437e4e69f00485e8da209c8ece3dccf"
server_key_iv: "dc6e7e6086c2bf881cea89f9cc5403a4"
server_key_version: "encryptionkey@0"
I20251025 08:02:59.975359 4663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "2bb6c9ea3e8c44fbae8e9cf7da26c365"
format_stamp: "Formatted at 2025-10-25 08:02:59 on dist-test-slave-lq8l"
server_key: "6437e4e69f00485e8da209c8ece3dccf"
server_key_iv: "dc6e7e6086c2bf881cea89f9cc5403a4"
server_key_version: "encryptionkey@0"
I20251025 08:02:59.978483 4663 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251025 08:02:59.980522 4678 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:02:59.981490 4663 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:02:59.981611 4663 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "2bb6c9ea3e8c44fbae8e9cf7da26c365"
format_stamp: "Formatted at 2025-10-25 08:02:59 on dist-test-slave-lq8l"
server_key: "6437e4e69f00485e8da209c8ece3dccf"
server_key_iv: "dc6e7e6086c2bf881cea89f9cc5403a4"
server_key_version: "encryptionkey@0"
I20251025 08:02:59.981696 4663 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:00.003314 4663 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:00.006217 4663 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:00.006408 4663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:00.006968 4663 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:00.007895 4663 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:00.007947 4663 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.008008 4663 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:00.008041 4663 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.017282 4663 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:34489
I20251025 08:03:00.017311 4791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:34489 every 8 connection(s)
I20251025 08:03:00.018208 4663 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379379, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:03:00.028054 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 4663
I20251025 08:03:00.028193 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:03:00.028426 28176 external_mini_cluster.cc:1468] Setting key 4e1dceccb52a6274a78823e2c6c9f6e5
I20251025 08:03:00.030093 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:00.020055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:38253 (local address 127.27.132.62:46479)
1025 08:03:00.020203 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:00.020207 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:00.020791 (+ 584us) server_negotiation.cc:408] Connection header received
1025 08:03:00.021606 (+ 815us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:00.021610 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:00.021672 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:00.021756 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:00.023449 (+ 1693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.023970 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.024602 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.024796 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.026900 (+ 2104us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:00.026919 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:00.026921 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:03:00.026944 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:00.028281 (+ 1337us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:00.028770 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:00.028774 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:00.028776 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:00.028821 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:00.029099 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:00.029102 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:00.029104 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:00.029247 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:00.029335 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:00.029812 (+ 477us) server_negotiation.cc:300] Negotiation successful
1025 08:03:00.029936 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251025 08:03:00.030707 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:00.020212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:38253)
1025 08:03:00.020639 (+ 427us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:00.020675 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:00.021399 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:00.021922 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:00.021931 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:00.022323 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:00.023277 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.023289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.024089 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.024092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:00.024491 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.024498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.024706 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.025314 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:00.025338 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:00.026750 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:00.028383 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:00.028389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:00.028399 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:00.028679 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:00.028900 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:00.028903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:00.028905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:00.029014 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:00.029345 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:00.029353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:00.029578 (+ 225us) client_negotiation.cc:769] Sending connection context
1025 08:03:00.029761 (+ 183us) client_negotiation.cc:240] Negotiation successful
1025 08:03:00.030018 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20251025 08:03:00.031731 4792 heartbeater.cc:344] Connected to a master server at 127.27.132.62:46479
I20251025 08:03:00.031965 4792 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:00.032429 4792 heartbeater.cc:507] Master 127.27.132.62:46479 requested a full tablet report, sending...
I20251025 08:03:00.033757 4599 ts_manager.cc:194] Registered new tserver with Master: 2bb6c9ea3e8c44fbae8e9cf7da26c365 (127.27.132.1:34489)
I20251025 08:03:00.034843 4599 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:38253
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:03:00.072139 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:46479
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:03:00.182356 4799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:00.182574 4799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:00.182631 4799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:00.185730 4799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:00.185792 4799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:00.185885 4799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:03:00.189929 4799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:00.190999 4799 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:00.191860 4799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:00.198156 4804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:00.198207 4807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:00.198307 4799 server_base.cc:1047] running on GCE node
W20251025 08:03:00.198156 4805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:00.198732 4799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:00.199231 4799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:00.200376 4799 hybrid_clock.cc:648] HybridClock initialized: now 1761379380200359 us; error 34 us; skew 500 ppm
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379380, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:03:00.202913 4799 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:03:00.203969 4799 webserver.cc:492] Webserver started at http://127.27.132.2:36141/ using document root <none> and password file <none>
I20251025 08:03:00.204499 4799 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:00.204564 4799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:00.204751 4799 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:00.206440 4799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "6bd7267cf64d407c84adb713b8dc68bd"
format_stamp: "Formatted at 2025-10-25 08:03:00 on dist-test-slave-lq8l"
server_key: "b88352d5a5450f5cda86a9a6a3faeeaa"
server_key_iv: "e656e5407ea1e8ba291db07fbfd6b90c"
server_key_version: "encryptionkey@0"
I20251025 08:03:00.206911 4799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "6bd7267cf64d407c84adb713b8dc68bd"
format_stamp: "Formatted at 2025-10-25 08:03:00 on dist-test-slave-lq8l"
server_key: "b88352d5a5450f5cda86a9a6a3faeeaa"
server_key_iv: "e656e5407ea1e8ba291db07fbfd6b90c"
server_key_version: "encryptionkey@0"
I20251025 08:03:00.210296 4799 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20251025 08:03:00.212570 4814 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.213568 4799 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:03:00.213678 4799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "6bd7267cf64d407c84adb713b8dc68bd"
format_stamp: "Formatted at 2025-10-25 08:03:00 on dist-test-slave-lq8l"
server_key: "b88352d5a5450f5cda86a9a6a3faeeaa"
server_key_iv: "e656e5407ea1e8ba291db07fbfd6b90c"
server_key_version: "encryptionkey@0"
I20251025 08:03:00.213771 4799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:00.230746 4799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:00.233739 4799 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:00.233927 4799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:00.234452 4799 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:00.235306 4799 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:00.235404 4799 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.235450 4799 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:00.235473 4799 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.244385 4799 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:45945
I20251025 08:03:00.244452 4927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:45945 every 8 connection(s)
I20251025 08:03:00.245296 4799 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:03:00.247967 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 4799
I20251025 08:03:00.248103 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:03:00.248342 28176 external_mini_cluster.cc:1468] Setting key 92a978ff8f6f2576f0ac838c89d0c480
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379380, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
I20251025 08:03:00.257643 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:00.246973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:51573 (local address 127.27.132.62:46479)
1025 08:03:00.247117 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:00.247122 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:00.247843 (+ 721us) server_negotiation.cc:408] Connection header received
1025 08:03:00.248692 (+ 849us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:00.248695 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:00.248756 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:00.248838 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:00.250543 (+ 1705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.251079 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.251776 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.251976 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.254366 (+ 2390us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:00.254383 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:00.254386 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:03:00.254410 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:00.255824 (+ 1414us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:00.256295 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:00.256298 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:00.256299 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:00.256342 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:00.256599 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:00.256602 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:00.256606 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:00.256752 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:00.256844 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:00.257340 (+ 496us) server_negotiation.cc:300] Negotiation successful
1025 08:03:00.257463 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251025 08:03:00.258323 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:00.247241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:51573)
1025 08:03:00.247704 (+ 463us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:00.247740 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:00.248505 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:00.248996 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:00.249004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:00.249377 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:00.250399 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.250427 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.251205 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.251208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:00.251668 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.251676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.251895 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.252531 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:00.252556 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:00.254222 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:00.255946 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:00.255952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:00.255964 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:00.256195 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:00.256424 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:00.256427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:00.256429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:00.256511 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:00.256859 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:00.256867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:00.257097 (+ 230us) client_negotiation.cc:769] Sending connection context
1025 08:03:00.257293 (+ 196us) client_negotiation.cc:240] Negotiation successful
1025 08:03:00.257564 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
I20251025 08:03:00.259517 4928 heartbeater.cc:344] Connected to a master server at 127.27.132.62:46479
I20251025 08:03:00.259748 4928 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:00.260257 4928 heartbeater.cc:507] Master 127.27.132.62:46479 requested a full tablet report, sending...
I20251025 08:03:00.261221 4599 ts_manager.cc:194] Registered new tserver with Master: 6bd7267cf64d407c84adb713b8dc68bd (127.27.132.2:45945)
I20251025 08:03:00.261765 4599 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:51573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:03:00.294571 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:46479
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:03:00.402622 4935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:00.402853 4935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:00.402912 4935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:00.406123 4935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:00.406190 4935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:00.406270 4935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:03:00.410475 4935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:00.411567 4935 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:00.412375 4935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:00.418833 4943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:00.418830 4940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:00.418984 4935 server_base.cc:1047] running on GCE node
W20251025 08:03:00.419098 4941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:00.419489 4935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:00.420029 4935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:00.421172 4935 hybrid_clock.cc:648] HybridClock initialized: now 1761379380421161 us; error 26 us; skew 500 ppm
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379380, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:03:00.423789 4935 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:03:00.424822 4935 webserver.cc:492] Webserver started at http://127.27.132.3:45887/ using document root <none> and password file <none>
I20251025 08:03:00.425375 4935 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:00.425432 4935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:00.425621 4935 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:00.427376 4935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "9fd81f58ff7b4cce8bbd6e4816464a29"
format_stamp: "Formatted at 2025-10-25 08:03:00 on dist-test-slave-lq8l"
server_key: "41f039fc00c30c12b598e8753d82cc1e"
server_key_iv: "98b416b863c63789c40e587ccc606f21"
server_key_version: "encryptionkey@0"
I20251025 08:03:00.427865 4935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "9fd81f58ff7b4cce8bbd6e4816464a29"
format_stamp: "Formatted at 2025-10-25 08:03:00 on dist-test-slave-lq8l"
server_key: "41f039fc00c30c12b598e8753d82cc1e"
server_key_iv: "98b416b863c63789c40e587ccc606f21"
server_key_version: "encryptionkey@0"
I20251025 08:03:00.431217 4935 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20251025 08:03:00.433503 4950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.434482 4935 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:03:00.434593 4935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "9fd81f58ff7b4cce8bbd6e4816464a29"
format_stamp: "Formatted at 2025-10-25 08:03:00 on dist-test-slave-lq8l"
server_key: "41f039fc00c30c12b598e8753d82cc1e"
server_key_iv: "98b416b863c63789c40e587ccc606f21"
server_key_version: "encryptionkey@0"
I20251025 08:03:00.434695 4935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:00.450302 4935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:00.453341 4935 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:00.453552 4935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:00.454133 4935 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:00.455036 4935 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:00.455088 4935 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.455153 4935 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:00.455181 4935 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.464534 4935 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:43029
I20251025 08:03:00.464572 5063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:43029 every 8 connection(s)
I20251025 08:03:00.465510 4935 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:03:00.470615 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 4935
I20251025 08:03:00.470715 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:03:00.470966 28176 external_mini_cluster.cc:1468] Setting key 6bda13d62ae926389fb2c25f17a8e634
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379380, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:03:00.477988 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:00.467296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:52905 (local address 127.27.132.62:46479)
1025 08:03:00.467456 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:00.467461 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:00.468205 (+ 744us) server_negotiation.cc:408] Connection header received
1025 08:03:00.469155 (+ 950us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:00.469158 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:00.469207 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:00.469282 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:00.471105 (+ 1823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.471805 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.472636 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.472815 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.475050 (+ 2235us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:00.475065 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:00.475068 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:03:00.475095 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:00.476416 (+ 1321us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:00.476879 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:00.476882 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:00.476883 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:00.476924 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:00.477129 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:00.477131 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:00.477133 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:00.477271 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:00.477340 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:00.477772 (+ 432us) server_negotiation.cc:300] Negotiation successful
1025 08:03:00.477865 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251025 08:03:00.478701 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:00.467592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:52905)
1025 08:03:00.468051 (+ 459us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:00.468089 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:00.468955 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:00.469451 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:00.469459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:00.469867 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:00.470914 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.470930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.471929 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.471933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:00.472508 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.472519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.472702 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.473389 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:00.473414 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:00.474879 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:00.476527 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:00.476532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:00.476542 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:00.476796 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:00.476988 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:00.476990 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:00.476992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:00.477069 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:00.477341 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:00.477345 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:00.477574 (+ 229us) client_negotiation.cc:769] Sending connection context
1025 08:03:00.477744 (+ 170us) client_negotiation.cc:240] Negotiation successful
1025 08:03:00.477972 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":130,"threads_started":1}
I20251025 08:03:00.479805 5064 heartbeater.cc:344] Connected to a master server at 127.27.132.62:46479
I20251025 08:03:00.480054 5064 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:00.480578 5064 heartbeater.cc:507] Master 127.27.132.62:46479 requested a full tablet report, sending...
I20251025 08:03:00.481560 4599 ts_manager.cc:194] Registered new tserver with Master: 9fd81f58ff7b4cce8bbd6e4816464a29 (127.27.132.3:43029)
I20251025 08:03:00.482120 4599 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:52905
I20251025 08:03:00.484884 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.1.keytab.
Entry for principal oryx/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.1.keytab.
WARNING: no policy specified for oryx/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.2.keytab.
Entry for principal oryx/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.2.keytab.
WARNING: no policy specified for oryx/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.3.keytab.
Entry for principal oryx/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.3.keytab.
I20251025 08:03:00.552160 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 4663
I20251025 08:03:00.557735 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 4799
I20251025 08:03:00.563386 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 4935
I20251025 08:03:00.568765 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 4567
I20251025 08:03:00.575210 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:46479
--webserver_interface=127.27.132.62
--webserver_port=38231
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:46479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:03:00.682595 5073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:00.682852 5073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:00.682909 5073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:00.686616 5073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:03:00.686692 5073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:00.686717 5073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:00.686735 5073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:03:00.686753 5073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:03:00.691642 5073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:46479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=38231
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:00.692797 5073 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:00.693677 5073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:00.699795 5079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:00.699841 5078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:00.700150 5073 server_base.cc:1047] running on GCE node
W20251025 08:03:00.700327 5081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:00.700865 5073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:00.701857 5073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:00.703027 5073 hybrid_clock.cc:648] HybridClock initialized: now 1761379380703009 us; error 36 us; skew 500 ppm
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379380, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:03:00.705951 5073 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:03:00.707103 5073 webserver.cc:492] Webserver started at http://127.27.132.62:38231/ using document root <none> and password file <none>
I20251025 08:03:00.707727 5073 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:00.707813 5073 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:00.711658 5073 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:03:00.713817 5088 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.714901 5073 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:03:00.715034 5073 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "c834ec353ab64e6f9769f445da9665a6"
format_stamp: "Formatted at 2025-10-25 08:02:59 on dist-test-slave-lq8l"
server_key: "5764bb17f448a933ed88d3e6d3c74589"
server_key_iv: "02490f4689c8e4fc6540ab9afa993fb0"
server_key_version: "encryptionkey@0"
I20251025 08:03:00.715545 5073 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:00.732213 5073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:00.735188 5073 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:00.735407 5073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:00.743180 5140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:46479 every 8 connection(s)
I20251025 08:03:00.743185 5073 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:46479
I20251025 08:03:00.744367 5073 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:03:00.749127 5141 sys_catalog.cc:263] Verifying existing consensus state
I20251025 08:03:00.750823 5141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6: Bootstrap starting.
I20251025 08:03:00.751116 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 5073
I20251025 08:03:00.751689 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:34489
--local_ip_for_outbound_sockets=127.27.132.1
--tserver_master_addrs=127.27.132.62:46479
--webserver_port=42801
--webserver_interface=127.27.132.1
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.132.1
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251025 08:03:00.768096 5141 log.cc:826] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6: Log is configured to *not* fsync() on all Append() calls
I20251025 08:03:00.774406 5141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251025 08:03:00.774812 5141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6: Bootstrap complete.
I20251025 08:03:00.778335 5141 raft_consensus.cc:359] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } }
I20251025 08:03:00.778661 5141 raft_consensus.cc:740] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c834ec353ab64e6f9769f445da9665a6, State: Initialized, Role: FOLLOWER
I20251025 08:03:00.779172 5141 consensus_queue.cc:260] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } }
I20251025 08:03:00.779294 5141 raft_consensus.cc:399] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:03:00.779389 5141 raft_consensus.cc:493] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:03:00.779485 5141 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 1 FOLLOWER]: Advancing to term 2
I20251025 08:03:00.781064 5141 raft_consensus.cc:515] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } }
I20251025 08:03:00.781402 5141 leader_election.cc:304] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c834ec353ab64e6f9769f445da9665a6; no voters:
I20251025 08:03:00.781720 5141 leader_election.cc:290] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251025 08:03:00.781814 5145 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 08:03:00.781999 5145 raft_consensus.cc:697] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [term 2 LEADER]: Becoming Leader. State: Replica: c834ec353ab64e6f9769f445da9665a6, State: Running, Role: LEADER
I20251025 08:03:00.782287 5145 consensus_queue.cc:237] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } }
I20251025 08:03:00.782893 5141 sys_catalog.cc:565] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:03:00.783604 5146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c834ec353ab64e6f9769f445da9665a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } } }
I20251025 08:03:00.783721 5146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [sys.catalog]: This master's current role is: LEADER
I20251025 08:03:00.784029 5147 sys_catalog.cc:455] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c834ec353ab64e6f9769f445da9665a6. Latest consensus state: current_term: 2 leader_uuid: "c834ec353ab64e6f9769f445da9665a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c834ec353ab64e6f9769f445da9665a6" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 46479 } } }
I20251025 08:03:00.784117 5147 sys_catalog.cc:458] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6 [sys.catalog]: This master's current role is: LEADER
I20251025 08:03:00.784547 5150 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:03:00.787700 5150 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:03:00.788367 5150 catalog_manager.cc:1269] Loaded cluster ID: ec43db44a2144dda81442a316bd8f163
I20251025 08:03:00.788431 5150 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:03:00.790904 5150 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:03:00.791687 5150 catalog_manager.cc:6033] T 00000000000000000000000000000000 P c834ec353ab64e6f9769f445da9665a6: Loaded TSK: 0
I20251025 08:03:00.792943 5150 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251025 08:03:00.865620 5143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:00.865871 5143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:00.865963 5143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:00.869253 5143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:00.869329 5143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:00.869449 5143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:03:00.873719 5143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.1.keytab
--principal=oryx/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:34489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=42801
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:00.874881 5143 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:00.875820 5143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:00.882536 5170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:00.882419 5168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:00.882417 5167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:00.882927 5143 server_base.cc:1047] running on GCE node
I20251025 08:03:00.883297 5143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:00.883908 5143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:00.885133 5143 hybrid_clock.cc:648] HybridClock initialized: now 1761379380885114 us; error 37 us; skew 500 ppm
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379380, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:03:00.887950 5143 init.cc:377] Logged in from keytab as oryx/127.27.132.1@KRBTEST.COM (short username oryx)
I20251025 08:03:00.889268 5143 webserver.cc:492] Webserver started at http://127.27.132.1:42801/ using document root <none> and password file <none>
I20251025 08:03:00.889964 5143 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:00.890048 5143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:00.894579 5143 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20251025 08:03:00.896968 5177 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.898172 5143 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:03:00.898301 5143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "2bb6c9ea3e8c44fbae8e9cf7da26c365"
format_stamp: "Formatted at 2025-10-25 08:02:59 on dist-test-slave-lq8l"
server_key: "6437e4e69f00485e8da209c8ece3dccf"
server_key_iv: "dc6e7e6086c2bf881cea89f9cc5403a4"
server_key_version: "encryptionkey@0"
I20251025 08:03:00.898854 5143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:00.914160 5143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:00.918115 5143 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:00.918354 5143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:00.919075 5143 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:00.920379 5143 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:00.920449 5143 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.920519 5143 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:00.920554 5143 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:00.933035 5143 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:34489
I20251025 08:03:00.933053 5290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:34489 every 8 connection(s)
I20251025 08:03:00.934430 5143 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:03:00.938305 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 5143
I20251025 08:03:00.938812 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:45945
--local_ip_for_outbound_sockets=127.27.132.2
--tserver_master_addrs=127.27.132.62:46479
--webserver_port=36141
--webserver_interface=127.27.132.2
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.132.2
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:00.949028 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:00.936871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:46565)
1025 08:03:00.937338 (+ 467us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:00.937373 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:00.938216 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:00.939663 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:00.939677 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:00.940264 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:00.941251 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.941266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.942949 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.942956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:00.943709 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.943724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.944005 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.944927 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:00.944973 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:00.948132 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":148,"threads_started":1}
W20251025 08:03:00.949035 5293 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:00.936686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:46565 (local address 127.27.132.62:46479)
1025 08:03:00.937088 (+ 402us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:00.937098 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:00.937486 (+ 388us) server_negotiation.cc:408] Connection header received
1025 08:03:00.938772 (+ 1286us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:00.938792 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:00.939158 (+ 366us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:00.939556 (+ 398us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:00.941430 (+ 1874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.942750 (+ 1320us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.943878 (+ 1128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.944167 (+ 289us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.948373 (+ 4206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:46565: BlockingRecv error: recv got EOF from 127.27.132.1:46565 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":105,"threads_started":1}
W20251025 08:03:00.949918 5291 heartbeater.cc:646] Failed to heartbeat to 127.27.132.62:46479 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.132.62:46479: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:00.956205 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:00.950379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:33119)
1025 08:03:00.950581 (+ 202us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:00.950599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:00.950681 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:00.950991 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:00.950994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:00.951168 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:00.951463 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.951472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.952562 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.952566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:00.953154 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.953164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.953283 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.953970 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:00.953986 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:00.956075 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20251025 08:03:00.956334 5293 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:00.950485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:33119 (local address 127.27.132.62:46479)
1025 08:03:00.950627 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:00.950634 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:00.950651 (+ 17us) server_negotiation.cc:408] Connection header received
1025 08:03:00.950796 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:00.950802 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:00.950868 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:00.950972 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:00.951595 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.952418 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.953296 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.953516 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.956228 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:33119: BlockingRecv error: recv got EOF from 127.27.132.1:33119 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:00 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:00.962440 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:00.956994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:60923)
1025 08:03:00.957166 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:00.957180 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:00.957269 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:00.957524 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:00.957527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:00.957690 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:00.957903 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.957911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.958864 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.958868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:00.959466 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:00.959478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.959597 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.960185 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:00.960203 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:00.962263 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20251025 08:03:00.962510 5293 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:00.957071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:60923 (local address 127.27.132.62:46479)
1025 08:03:00.957179 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:00.957184 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:00.957227 (+ 43us) server_negotiation.cc:408] Connection header received
1025 08:03:00.957360 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:00.957363 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:00.957411 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:00.957495 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:00.958024 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.958729 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:00.959616 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:00.960156 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:00.962399 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:60923: BlockingRecv error: recv got EOF from 127.27.132.1:60923 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20251025 08:03:00.962980 5291 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251025 08:03:01.054872 5295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:01.055112 5295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:01.055205 5295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:01.058575 5295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:01.058663 5295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:01.058787 5295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:03:01.063128 5295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.2.keytab
--principal=oryx/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:45945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=36141
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:01.064311 5295 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:01.065201 5295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:01.072572 5301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:01.072572 5303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:01.072829 5295 server_base.cc:1047] running on GCE node
W20251025 08:03:01.072886 5300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:01.073247 5295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:01.073805 5295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:01.074954 5295 hybrid_clock.cc:648] HybridClock initialized: now 1761379381074943 us; error 29 us; skew 500 ppm
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379381, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:03:01.077539 5295 init.cc:377] Logged in from keytab as oryx/127.27.132.2@KRBTEST.COM (short username oryx)
I20251025 08:03:01.078545 5295 webserver.cc:492] Webserver started at http://127.27.132.2:36141/ using document root <none> and password file <none>
I20251025 08:03:01.079084 5295 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:01.079154 5295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:01.082564 5295 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:03:01.084527 5310 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:01.085578 5295 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:03:01.085675 5295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "6bd7267cf64d407c84adb713b8dc68bd"
format_stamp: "Formatted at 2025-10-25 08:03:00 on dist-test-slave-lq8l"
server_key: "b88352d5a5450f5cda86a9a6a3faeeaa"
server_key_iv: "e656e5407ea1e8ba291db07fbfd6b90c"
server_key_version: "encryptionkey@0"
I20251025 08:03:01.086081 5295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:01.121783 5295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:01.125149 5295 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:01.125329 5295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:01.125901 5295 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:01.126772 5295 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:01.126822 5295 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:01.126865 5295 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:01.126901 5295 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:01.136247 5295 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:45945
I20251025 08:03:01.136287 5423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:45945 every 8 connection(s)
I20251025 08:03:01.137281 5295 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:03:01.146159 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 5295
I20251025 08:03:01.146709 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:43029
--local_ip_for_outbound_sockets=127.27.132.3
--tserver_master_addrs=127.27.132.62:46479
--webserver_port=45887
--webserver_interface=127.27.132.3
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.132.3
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
W20251025 08:03:01.149979 5293 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:01.139157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:38957 (local address 127.27.132.62:46479)
1025 08:03:01.139309 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:01.139314 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:01.140146 (+ 832us) server_negotiation.cc:408] Connection header received
1025 08:03:01.141286 (+ 1140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:01.141292 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:01.141367 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:01.141473 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:01.143768 (+ 2295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.144371 (+ 603us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.145359 (+ 988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.145674 (+ 315us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.149845 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:38957: BlockingRecv error: recv got EOF from 127.27.132.2:38957 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20251025 08:03:01.150631 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:01.139479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:38957)
1025 08:03:01.139956 (+ 477us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:01.140024 (+ 68us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:01.141034 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:01.141681 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:01.141692 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:01.142252 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:01.143589 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.143608 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.144570 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.144574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:01.145193 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.145204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.145450 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.146183 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:01.146252 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:01.149717 (+ 3465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":116,"threads_started":1}
W20251025 08:03:01.151149 5424 heartbeater.cc:646] Failed to heartbeat to 127.27.132.62:46479 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.132.62:46479: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:01.156755 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:01.151651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:60767)
1025 08:03:01.151917 (+ 266us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:01.151932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:01.152028 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:01.152276 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:01.152279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:01.152454 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:01.152649 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.152655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.153373 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.153377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:01.153762 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.153770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.153879 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.154507 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:01.154521 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:01.156594 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194}
W20251025 08:03:01.156863 5293 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:01.151789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:60767 (local address 127.27.132.62:46479)
1025 08:03:01.151904 (+ 115us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:01.151909 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:01.151979 (+ 70us) server_negotiation.cc:408] Connection header received
1025 08:03:01.152132 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:01.152134 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:01.152178 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:01.152248 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:01.152775 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.153258 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.153874 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.154080 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.156736 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:60767: BlockingRecv error: recv got EOF from 127.27.132.2:60767 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:01.162290 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:01.157462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:40453)
1025 08:03:01.157697 (+ 235us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:01.157709 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:01.157786 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:01.158043 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:01.158045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:01.158173 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:01.158332 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.158337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.159432 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.159435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:01.159803 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.159810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.159901 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.160405 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:01.160418 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:01.162192 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168}
W20251025 08:03:01.162454 5293 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:01.157577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:40453 (local address 127.27.132.62:46479)
1025 08:03:01.157732 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:01.157740 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:01.157759 (+ 19us) server_negotiation.cc:408] Connection header received
1025 08:03:01.157864 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:01.157867 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:01.157931 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:01.158036 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:01.158438 (+ 402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.159248 (+ 810us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.159920 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.160115 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.162341 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:40453: BlockingRecv error: recv got EOF from 127.27.132.2:40453 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20251025 08:03:01.162606 5424 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251025 08:03:01.257061 5427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:01.257313 5427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:01.257409 5427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:01.260811 5427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:01.260898 5427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:01.261024 5427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:03:01.265309 5427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/krb5kdc/oryx_127.27.132.3.keytab
--principal=oryx/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:43029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=45887
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:01.266479 5427 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:01.267391 5427 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:01.273945 5433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:01.273938 5432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:01.274129 5427 server_base.cc:1047] running on GCE node
W20251025 08:03:01.273941 5435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:01.274580 5427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:01.275101 5427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:01.276252 5427 hybrid_clock.cc:648] HybridClock initialized: now 1761379381276218 us; error 56 us; skew 500 ppm
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379381, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:03:01.278915 5427 init.cc:377] Logged in from keytab as oryx/127.27.132.3@KRBTEST.COM (short username oryx)
I20251025 08:03:01.279979 5427 webserver.cc:492] Webserver started at http://127.27.132.3:45887/ using document root <none> and password file <none>
I20251025 08:03:01.280520 5427 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:01.280591 5427 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:01.284148 5427 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:03:01.286216 5442 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:01.287217 5427 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:03:01.287362 5427 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "9fd81f58ff7b4cce8bbd6e4816464a29"
format_stamp: "Formatted at 2025-10-25 08:03:00 on dist-test-slave-lq8l"
server_key: "41f039fc00c30c12b598e8753d82cc1e"
server_key_iv: "98b416b863c63789c40e587ccc606f21"
server_key_version: "encryptionkey@0"
I20251025 08:03:01.287786 5427 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:01.307385 5427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:01.310125 5427 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:01.310307 5427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:01.310866 5427 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:01.311738 5427 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:01.311794 5427 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:01.311857 5427 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:01.311885 5427 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:01.321072 5427 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:43029
I20251025 08:03:01.321120 5555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:43029 every 8 connection(s)
I20251025 08:03:01.322053 5427 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:03:01.324152 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 5427
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
W20251025 08:03:01.332758 5293 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:01.323690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:35647 (local address 127.27.132.62:46479)
1025 08:03:01.323818 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:01.323821 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:01.324588 (+ 767us) server_negotiation.cc:408] Connection header received
1025 08:03:01.325908 (+ 1320us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:01.325911 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:01.325957 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:01.326035 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:01.327953 (+ 1918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.328624 (+ 671us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.329339 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.329464 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.332639 (+ 3175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:35647: BlockingRecv error: recv got EOF from 127.27.132.3:35647 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20251025 08:03:01.332996 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:01.325055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.27.132.62:46479)
1025 08:03:01.325317 (+ 262us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:01.325323 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:01.325338 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:03:01.325383 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:01.325386 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:01.325435 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:01.325531 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:01.326320 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.327097 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.327751 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.328244 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.329089 (+ 845us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:01.329111 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:01.329125 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:03:01.329173 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:01.331144 (+ 1971us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:01.331607 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:01.331615 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:01.331622 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:01.331715 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:01.331964 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:01.331966 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:01.331967 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:01.332205 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:01.332336 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:01.332625 (+ 289us) server_negotiation.cc:300] Negotiation successful
1025 08:03:01.332860 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20251025 08:03:01.333333 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:01.323939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:35647)
1025 08:03:01.324431 (+ 492us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:01.324466 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:01.325729 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:01.326197 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:01.326204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:01.326694 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:01.327794 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.327809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.328760 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.328763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:01.329220 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.329232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.329450 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.329945 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:01.329981 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:01.332506 (+ 2525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
W20251025 08:03:01.333751 5556 heartbeater.cc:646] Failed to heartbeat to 127.27.132.62:46479 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.132.62:46479: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:01.338514 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:01.334164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:37643)
1025 08:03:01.334276 (+ 112us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:01.334292 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:01.334367 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:01.334585 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:01.334588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:01.334730 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:01.334940 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.334949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.335986 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.335989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:01.336354 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.336361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.336463 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.336923 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:01.336935 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:01.338380 (+ 1445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20251025 08:03:01.338608 5560 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:01.334230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:37643 (local address 127.27.132.62:46479)
1025 08:03:01.334349 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:01.334352 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:01.334364 (+ 12us) server_negotiation.cc:408] Connection header received
1025 08:03:01.334455 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:01.334458 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:01.334497 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:01.334565 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:01.335038 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.335876 (+ 838us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.336458 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.336657 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.338502 (+ 1845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:37643: BlockingRecv error: recv got EOF from 127.27.132.3:37643 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:01.342943 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:01.339023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:33181)
1025 08:03:01.339121 (+ 98us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:01.339132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:01.339204 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:01.339490 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:01.339493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:01.339636 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:01.339769 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.339774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.340418 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.340421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:01.340802 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.340809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.340910 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.341342 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:01.341354 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:01.342852 (+ 1498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20251025 08:03:01.343088 5560 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:01.339112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:33181 (local address 127.27.132.62:46479)
1025 08:03:01.339270 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:01.339276 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:01.339287 (+ 11us) server_negotiation.cc:408] Connection header received
1025 08:03:01.339317 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:01.339320 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:01.339397 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:01.339470 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:01.339882 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.340318 (+ 436us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.340907 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.341097 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.342965 (+ 1868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:33181: BlockingRecv error: recv got EOF from 127.27.132.3:33181 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20251025 08:03:01.343132 5556 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:01 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:01.968572 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:01.963650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:51009)
1025 08:03:01.963932 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:01.963945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:01.964038 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:01.964300 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:01.964302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:01.964491 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:01.964723 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.964729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.965442 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.965445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:01.965845 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:01.965852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.965938 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.966501 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:01.966515 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:01.968401 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
W20251025 08:03:01.968689 5562 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:01.963747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:51009 (local address 127.27.132.62:46479)
1025 08:03:01.963991 (+ 244us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:01.963994 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:01.964007 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:03:01.964134 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:01.964137 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:01.964184 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:01.964263 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:01.964848 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.965357 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:01.965936 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:01.966068 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:01.968558 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:51009: BlockingRecv error: recv got EOF from 127.27.132.1:51009 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
Oct 25 08:03:02 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:02 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:02.167913 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:02.163180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:49891)
1025 08:03:02.163467 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:02.163481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:02.163595 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:02.163819 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:02.163821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:02.164002 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:02.164216 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:02.164222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.164931 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:02.164934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:02.165338 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:02.165345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.165447 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:02.166012 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:02.166024 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:02.167789 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":74,"threads_started":1}
W20251025 08:03:02.168035 5562 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:02.163298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:49891 (local address 127.27.132.62:46479)
1025 08:03:02.163447 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:02.163451 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:02.163576 (+ 125us) server_negotiation.cc:408] Connection header received
1025 08:03:02.163671 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:02.163673 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:02.163723 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:02.163795 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:02.164307 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.164825 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:02.165429 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.165572 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:02.167932 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:49891: BlockingRecv error: recv got EOF from 127.27.132.2:49891 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Oct 25 08:03:02 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:02 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:02.348626 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:02.343730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:56751)
1025 08:03:02.344001 (+ 271us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:02.344014 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:02.344104 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:02.344328 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:02.344330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:02.344526 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:02.344739 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:02.344744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.345445 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:02.345448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:02.345842 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:02.345849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.345933 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:02.346518 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:02.346532 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:02.348459 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":69,"threads_started":1}
W20251025 08:03:02.348731 5562 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:02.343957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:56751 (local address 127.27.132.62:46479)
1025 08:03:02.344091 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:02.344094 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:02.344107 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:03:02.344181 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:02.344183 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:02.344227 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:02.344300 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:02.344872 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.345359 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:02.345933 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.346059 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:02.348624 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:56751: BlockingRecv error: recv got EOF from 127.27.132.3:56751 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Oct 25 08:03:02 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:02 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:02.974257 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:02.969400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:35933)
1025 08:03:02.969669 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:02.969683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:02.969773 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:02.970005 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:02.970007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:02.970195 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:02.970392 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:02.970399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.971168 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:02.971171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:02.971591 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:02.971599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.971694 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:02.972265 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:02.972279 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:02.974102 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
W20251025 08:03:02.974385 5566 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:02.969471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:35933 (local address 127.27.132.62:46479)
1025 08:03:02.969742 (+ 271us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:02.969746 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:02.969760 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:03:02.969853 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:02.969855 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:02.969900 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:02.969979 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:02.970559 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.971078 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:02.971712 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:02.971862 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:02.974257 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:35933: BlockingRecv error: recv got EOF from 127.27.132.1:35933 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":99,"threads_started":1}
Oct 25 08:03:03 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:03 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:03.173868 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:03.168713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:49175)
1025 08:03:03.168963 (+ 250us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:03.168976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:03.169077 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:03.169340 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:03.169343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:03.169518 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:03.169708 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:03.169714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.170950 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:03.170953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:03.171375 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:03.171384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.171475 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:03.172027 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:03.172040 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:03.173753 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
W20251025 08:03:03.174115 5566 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:03.168803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:49175 (local address 127.27.132.62:46479)
1025 08:03:03.168979 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:03.168986 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:03.169049 (+ 63us) server_negotiation.cc:408] Connection header received
1025 08:03:03.169161 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:03.169166 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:03.169233 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:03.169334 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:03.169825 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.170833 (+ 1008us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:03.171500 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.171750 (+ 250us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:03.173950 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:49175: BlockingRecv error: recv got EOF from 127.27.132.2:49175 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Oct 25 08:03:03 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:03 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:03.354789 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:03.349572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:57157)
1025 08:03:03.349895 (+ 323us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:03.349908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:03.350015 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:03.350263 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:03.350265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:03.350473 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:03.350695 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:03.350701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.351455 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:03.351458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:03.351860 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:03.351867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.351953 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:03.352560 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:03.352573 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:03.354650 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":96,"threads_started":1}
W20251025 08:03:03.354871 5566 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:03.349758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:57157 (local address 127.27.132.62:46479)
1025 08:03:03.349897 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:03.349901 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:03.350016 (+ 115us) server_negotiation.cc:408] Connection header received
1025 08:03:03.350089 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:03.350091 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:03.350146 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:03.350226 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:03.350826 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.351360 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:03.351951 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.352083 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:03.354767 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:57157: BlockingRecv error: recv got EOF from 127.27.132.3:57157 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Oct 25 08:03:03 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:03 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:03.980324 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:03.975183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:35399)
1025 08:03:03.975519 (+ 336us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:03.975539 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:03.975653 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:03.975975 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:03.975977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:03.976163 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:03.976360 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:03.976367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.977144 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:03.977146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:03.977519 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:03.977525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.977609 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:03.978262 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:03.978277 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:03.980167 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":123,"threads_started":1}
W20251025 08:03:03.980435 5570 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:03.975313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:35399 (local address 127.27.132.62:46479)
1025 08:03:03.975608 (+ 295us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:03.975613 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:03.975632 (+ 19us) server_negotiation.cc:408] Connection header received
1025 08:03:03.975794 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:03.975798 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:03.975851 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:03.975937 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:03.976517 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.977047 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:03.977612 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:03.977747 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:03.980309 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:35399: BlockingRecv error: recv got EOF from 127.27.132.1:35399 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":65,"threads_started":1}
Oct 25 08:03:04 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:04 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:04.179749 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:04.174770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:45173)
1025 08:03:04.175049 (+ 279us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:04.175062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:04.175169 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:04.175435 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:04.175437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:04.175621 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:04.175820 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:04.175826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.176545 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:04.176549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:04.176932 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:04.176940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.177040 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:04.177888 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:04.177902 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:04.179634 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
W20251025 08:03:04.179908 5570 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:04.174885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:45173 (local address 127.27.132.62:46479)
1025 08:03:04.175015 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:04.175019 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:04.175128 (+ 109us) server_negotiation.cc:408] Connection header received
1025 08:03:04.175264 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:04.175267 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:04.175314 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:04.175399 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:04.175920 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.176418 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:04.177037 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.177166 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:04.179765 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:45173: BlockingRecv error: recv got EOF from 127.27.132.2:45173 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Oct 25 08:03:04 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:04 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:04.360771 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:04.355705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:57765)
1025 08:03:04.356029 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:04.356046 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:04.356174 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:04.356463 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:04.356467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:04.356680 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:04.356977 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:04.356986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.357705 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:04.357708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:04.358115 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:04.358123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.358225 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:04.358823 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:04.358836 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:04.360634 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
W20251025 08:03:04.360960 5570 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:04.355794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:57765 (local address 127.27.132.62:46479)
1025 08:03:04.355926 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:04.355931 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:04.356126 (+ 195us) server_negotiation.cc:408] Connection header received
1025 08:03:04.356261 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:04.356265 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:04.356330 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:04.356411 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:04.357085 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.357582 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:04.358216 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.358344 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:04.360815 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:57765: BlockingRecv error: recv got EOF from 127.27.132.3:57765 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Oct 25 08:03:04 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:04 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:04.991138 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:04.981315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:34239)
1025 08:03:04.981635 (+ 320us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:04.981650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:04.981749 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:04.982014 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:04.982016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:04.982199 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:04.982462 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:04.982472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.983392 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:04.983395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:04.983866 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:04.983875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.983971 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:04.989043 (+ 5072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:04.989059 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:04.991018 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":80,"threads_started":1}
W20251025 08:03:04.991366 5574 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:04.981546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:34239 (local address 127.27.132.62:46479)
1025 08:03:04.981785 (+ 239us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:04.981789 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:04.981822 (+ 33us) server_negotiation.cc:408] Connection header received
1025 08:03:04.981862 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:04.981864 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:04.981910 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:04.982006 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:04.982585 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.983217 (+ 632us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:04.983961 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:04.984131 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:04.991188 (+ 7057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:34239: BlockingRecv error: recv got EOF from 127.27.132.1:34239 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
Oct 25 08:03:05 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:05 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:05.185748 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:05.180600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:50163)
1025 08:03:05.180902 (+ 302us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:05.180917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:05.181048 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:05.181294 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:05.181297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:05.181494 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:05.181696 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:05.181706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.182649 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:05.182652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:05.183095 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:05.183105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.183200 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:05.183861 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:05.183875 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:05.185634 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
W20251025 08:03:05.185879 5574 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:05.180758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:50163 (local address 127.27.132.62:46479)
1025 08:03:05.180929 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:05.180933 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:05.181010 (+ 77us) server_negotiation.cc:408] Connection header received
1025 08:03:05.181137 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:05.181140 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:05.181189 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:05.181262 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:05.181853 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.182529 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:05.183195 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.183366 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:05.185764 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:50163: BlockingRecv error: recv got EOF from 127.27.132.2:50163 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Oct 25 08:03:05 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:05 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:05.366302 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:05.361650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:50979)
1025 08:03:05.361959 (+ 309us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:05.361972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:05.362089 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:05.362374 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:05.362376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:05.362559 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:05.362745 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:05.362751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.363433 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:05.363436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:05.363810 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:05.363817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.363903 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:05.364458 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:05.364471 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:05.366183 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
W20251025 08:03:05.366456 5574 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:05.361736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:50979 (local address 127.27.132.62:46479)
1025 08:03:05.361893 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:05.361896 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:05.362156 (+ 260us) server_negotiation.cc:408] Connection header received
1025 08:03:05.362202 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:05.362204 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:05.362256 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:05.362331 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:05.362835 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.363347 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:05.363903 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.364027 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:05.366332 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:50979: BlockingRecv error: recv got EOF from 127.27.132.3:50979 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Oct 25 08:03:05 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:05 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:05.997144 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:05.992156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:50291)
1025 08:03:05.992448 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:05.992463 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:05.992559 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:05.992821 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:05.992823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:05.993039 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:05.993234 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:05.993241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.993995 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:05.993999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:05.994381 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:05.994389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.994486 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:05.995070 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:05.995085 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:05.997018 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
W20251025 08:03:05.997380 5578 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:05.992263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:50291 (local address 127.27.132.62:46479)
1025 08:03:05.992508 (+ 245us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:05.992511 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:05.992526 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:03:05.992643 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:05.992646 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:05.992692 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:05.992788 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:05.993347 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.993885 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:05.994491 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:05.994632 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:05.997219 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:50291: BlockingRecv error: recv got EOF from 127.27.132.1:50291 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
Oct 25 08:03:06 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:06 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:06.192195 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:06.186628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:55309)
1025 08:03:06.186922 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:06.186937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:06.187029 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:06.187312 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:06.187314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:06.187575 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:06.187779 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:06.187786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:06.188553 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:06.188556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:06.188944 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:06.188952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:06.189110 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:06.189687 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:06.189701 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:06.191990 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
W20251025 08:03:06.192255 5578 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:06.186809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:55309 (local address 127.27.132.62:46479)
1025 08:03:06.186972 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:06.186976 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:06.186991 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:03:06.187155 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:06.187159 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:06.187210 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:06.187287 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:06.187897 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:06.188397 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:06.189085 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:06.189211 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:06.192148 (+ 2937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:55309: BlockingRecv error: recv got EOF from 127.27.132.2:55309 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Oct 25 08:03:06 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:06 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:06.372326 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:06.367213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:37839)
1025 08:03:06.367572 (+ 359us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:06.367591 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:06.367720 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:06.368000 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:06.368003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:06.368203 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:06.368390 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:06.368396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:06.369174 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:06.369177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:06.369558 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:06.369565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:06.369675 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:06.370258 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:06.370272 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:06.372214 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":119,"threads_started":1}
W20251025 08:03:06.372491 5578 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:06.367408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:37839 (local address 127.27.132.62:46479)
1025 08:03:06.367606 (+ 198us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:06.367613 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:06.367659 (+ 46us) server_negotiation.cc:408] Connection header received
1025 08:03:06.367830 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:06.367834 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:06.367884 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:06.367957 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:06.368526 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:06.369053 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:06.369676 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:06.369891 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:06.372351 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:37839: BlockingRecv error: recv got EOF from 127.27.132.3:37839 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Oct 25 08:03:06 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:06 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:07.002758 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:06.997971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:52647)
1025 08:03:06.998253 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:06.998267 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:06.998365 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:06.998635 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:06.998638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:06.998826 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:06.999024 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:06.999030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:06.999815 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:06.999818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:07.000193 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:07.000205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:07.000295 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:07.000849 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:07.000861 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:07.002636 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
W20251025 08:03:07.002948 5582 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:06.998092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:52647 (local address 127.27.132.62:46479)
1025 08:03:06.998332 (+ 240us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:06.998336 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:06.998351 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:03:06.998448 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:06.998451 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:06.998502 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:06.998580 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:06.999145 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:06.999709 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:07.000284 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:07.000408 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:07.002802 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:52647: BlockingRecv error: recv got EOF from 127.27.132.1:52647 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":62,"threads_started":1}
Oct 25 08:03:07 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:07 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:07.197872 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:07.193011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:57231)
1025 08:03:07.193286 (+ 275us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:07.193299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:07.193412 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:07.193674 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:07.193676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:07.193887 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:07.194086 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:07.194092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:07.194861 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:07.194864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:07.195243 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:07.195251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:07.195387 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:07.195969 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:07.195983 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:07.197749 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
W20251025 08:03:07.198032 5582 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:07.193061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:57231 (local address 127.27.132.62:46479)
1025 08:03:07.193195 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:07.193201 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:07.193366 (+ 165us) server_negotiation.cc:408] Connection header received
1025 08:03:07.193509 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:07.193513 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:07.193565 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:07.193637 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:07.194206 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:07.194744 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:07.195379 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:07.195594 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:07.197888 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:57231: BlockingRecv error: recv got EOF from 127.27.132.2:57231 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Oct 25 08:03:07 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:07 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:07.378039 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:07.373195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:35989)
1025 08:03:07.373486 (+ 291us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:07.373500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:07.373605 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:07.373839 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:07.373841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:07.374024 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:07.374221 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:07.374227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:07.374948 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:07.374952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:07.375390 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:07.375398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:07.375515 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:07.376098 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:07.376114 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:07.377923 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
W20251025 08:03:07.378216 5582 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:07.373311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:35989 (local address 127.27.132.62:46479)
1025 08:03:07.373490 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:07.373494 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:07.373569 (+ 75us) server_negotiation.cc:408] Connection header received
1025 08:03:07.373684 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:07.373687 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:07.373740 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:07.373814 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:07.374320 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:07.374833 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:07.375543 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:07.375678 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:07.378071 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:35989: BlockingRecv error: recv got EOF from 127.27.132.3:35989 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Oct 25 08:03:08 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:08 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:08.008410 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:08.003526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:33793)
1025 08:03:08.003861 (+ 335us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:08.003875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:08.003996 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:08.004218 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:08.004220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:08.004402 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:08.004614 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:08.004621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.005433 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:08.005437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:08.005834 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:08.005841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.005933 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:08.006491 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:08.006503 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:08.008292 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":86,"threads_started":1}
W20251025 08:03:08.008543 5586 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:08.003611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:33793 (local address 127.27.132.62:46479)
1025 08:03:08.003868 (+ 257us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:08.003871 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:08.003959 (+ 88us) server_negotiation.cc:408] Connection header received
1025 08:03:08.004071 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:08.004073 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:08.004118 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:08.004194 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:08.004725 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.005330 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:08.005928 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.006057 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:08.008424 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:33793: BlockingRecv error: recv got EOF from 127.27.132.1:33793 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
Oct 25 08:03:08 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:08 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:08.203810 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:08.198641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:59325)
1025 08:03:08.199011 (+ 370us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:08.199025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:08.199152 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:08.199456 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:08.199459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:08.199678 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:08.199885 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:08.199891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.200650 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:08.200653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:08.201035 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:08.201042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.201152 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:08.201799 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:08.201812 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:08.203632 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":96,"threads_started":1}
W20251025 08:03:08.203917 5586 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:08.198790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:59325 (local address 127.27.132.62:46479)
1025 08:03:08.198957 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:08.198962 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:08.199103 (+ 141us) server_negotiation.cc:408] Connection header received
1025 08:03:08.199254 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:08.199260 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:08.199306 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:08.199403 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:08.200018 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.200528 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:08.201156 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.201362 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:08.203795 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:59325: BlockingRecv error: recv got EOF from 127.27.132.2:59325 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Oct 25 08:03:08 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:08 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:08.384214 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:08.378985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:55783)
1025 08:03:08.379355 (+ 370us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:08.379368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:08.379491 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:08.379794 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:08.379797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:08.380001 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:08.380184 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:08.380189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.380940 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:08.380944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:08.381485 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:08.381496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.381611 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:08.382289 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:08.382302 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:08.384105 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
W20251025 08:03:08.384325 5586 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:08.379124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:55783 (local address 127.27.132.62:46479)
1025 08:03:08.379297 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:08.379302 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:08.379448 (+ 146us) server_negotiation.cc:408] Connection header received
1025 08:03:08.379611 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:08.379614 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:08.379683 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:08.379768 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:08.380330 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.380825 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:08.381623 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:08.381814 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:08.384226 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:55783: BlockingRecv error: recv got EOF from 127.27.132.3:55783 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Oct 25 08:03:09 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:09 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:09.014515 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:09.009469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:56029)
1025 08:03:09.009764 (+ 295us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:09.009778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:09.009875 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:09.010095 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:09.010098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:09.010277 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:09.010487 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:09.010494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.011251 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:09.011255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:09.011700 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:09.011710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.011822 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:09.012398 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:09.012411 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:09.014332 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
W20251025 08:03:09.014647 5590 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:09.009652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:56029 (local address 127.27.132.62:46479)
1025 08:03:09.009879 (+ 227us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:09.009883 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:09.009898 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:03:09.009939 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:09.009941 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:09.009986 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:09.010069 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:09.010619 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.011138 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:09.011816 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.011956 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:09.014484 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:56029: BlockingRecv error: recv got EOF from 127.27.132.1:56029 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
Oct 25 08:03:09 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:09 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:09.209751 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:09.204549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:44277)
1025 08:03:09.204875 (+ 326us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:09.204889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:09.205006 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:09.205264 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:09.205267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:09.205485 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:09.205681 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:09.205688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.206597 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:09.206601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:09.206987 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:09.206995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.207107 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:09.207735 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:09.207758 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:09.209633 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
W20251025 08:03:09.209980 5590 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:09.204649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:44277 (local address 127.27.132.62:46479)
1025 08:03:09.204807 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:09.204812 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:09.204964 (+ 152us) server_negotiation.cc:408] Connection header received
1025 08:03:09.205107 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:09.205112 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:09.205157 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:09.205231 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:09.205813 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.206474 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:09.207103 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.207305 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:09.209850 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:44277: BlockingRecv error: recv got EOF from 127.27.132.2:44277 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Oct 25 08:03:09 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:09 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:09.389875 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:09.384932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:45405)
1025 08:03:09.385203 (+ 271us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:09.385216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:09.385319 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:09.385565 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:09.385568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:09.385762 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:09.385954 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:09.385960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.386635 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:09.386638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:09.387024 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:09.387033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.387126 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:09.387739 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:09.387753 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:09.389713 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
W20251025 08:03:09.389997 5590 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:09.384996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:45405 (local address 127.27.132.62:46479)
1025 08:03:09.385131 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:09.385134 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:09.385282 (+ 148us) server_negotiation.cc:408] Connection header received
1025 08:03:09.385395 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:09.385397 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:09.385451 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:09.385527 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:09.386047 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.386531 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:09.387125 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:09.387249 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:09.389867 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:45405: BlockingRecv error: recv got EOF from 127.27.132.3:45405 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Oct 25 08:03:10 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:10 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:10.022992 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:10.015394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:52867)
1025 08:03:10.015802 (+ 408us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:10.015819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:10.015950 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:10.016238 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:10.016240 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:10.016498 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:10.016761 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:10.016769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.017489 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:10.017492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:10.017866 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:10.017873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.017962 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:10.021091 (+ 3129us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:10.021104 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:10.022880 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":124,"threads_started":1}
W20251025 08:03:10.023175 5594 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:10.015507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:52867 (local address 127.27.132.62:46479)
1025 08:03:10.015810 (+ 303us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:10.015815 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:10.015903 (+ 88us) server_negotiation.cc:408] Connection header received
1025 08:03:10.016052 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:10.016054 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:10.016101 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:10.016190 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:10.016868 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.017381 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:10.017965 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.018105 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:10.023023 (+ 4918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:52867: BlockingRecv error: recv got EOF from 127.27.132.1:52867 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
Oct 25 08:03:10 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:10 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:10.215571 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:10.210653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:35549)
1025 08:03:10.210932 (+ 279us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:10.210945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:10.211050 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:10.211276 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:10.211278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:10.211479 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:10.211673 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:10.211679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.212338 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:10.212341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:10.212709 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:10.212716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.212812 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:10.213388 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:10.213402 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:10.215397 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
W20251025 08:03:10.215636 5594 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:10.210757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:35549 (local address 127.27.132.62:46479)
1025 08:03:10.210923 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:10.210927 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:10.211015 (+ 88us) server_negotiation.cc:408] Connection header received
1025 08:03:10.211124 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:10.211127 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:10.211172 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:10.211239 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:10.211762 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.212251 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:10.212794 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.212917 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:10.215514 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:35549: BlockingRecv error: recv got EOF from 127.27.132.2:35549 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Oct 25 08:03:10 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:10 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:10.395471 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:10.390545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:56059)
1025 08:03:10.390848 (+ 303us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:10.390861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:10.390972 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:10.391227 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:10.391230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:10.391459 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:10.391665 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:10.391671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.392387 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:10.392391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:10.392798 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:10.392805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.392907 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:10.393512 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:10.393526 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:10.395310 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":87,"threads_started":1}
W20251025 08:03:10.395588 5594 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:10.390647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:56059 (local address 127.27.132.62:46479)
1025 08:03:10.390794 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:10.390799 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:10.390930 (+ 131us) server_negotiation.cc:408] Connection header received
1025 08:03:10.391064 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:10.391067 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:10.391117 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:10.391186 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:10.391779 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.392275 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:10.392908 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:10.393119 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:10.395472 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:56059: BlockingRecv error: recv got EOF from 127.27.132.3:56059 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Oct 25 08:03:11 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:11 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:11.028730 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:11.023856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:36689)
1025 08:03:11.024148 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:11.024162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:11.024249 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:11.024484 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:11.024486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:11.024673 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:11.024880 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:11.024886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.025615 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:11.025619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:11.026015 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:11.026023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.026127 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:11.026673 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:11.026686 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:11.028562 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
W20251025 08:03:11.028990 5598 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:11.023948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:36689 (local address 127.27.132.62:46479)
1025 08:03:11.024191 (+ 243us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:11.024194 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:11.024208 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:03:11.024335 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:11.024338 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:11.024381 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:11.024460 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:11.024975 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.025482 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:11.026107 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.026230 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:11.028688 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:36689: BlockingRecv error: recv got EOF from 127.27.132.1:36689 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":59,"threads_started":1}
Oct 25 08:03:11 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:11 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:11.221009 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:11.216386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:60951)
1025 08:03:11.216659 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:11.216673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:11.216771 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:11.216990 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:11.216992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:11.217173 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:11.217362 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:11.217367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.218101 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:11.218103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:11.218468 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:11.218473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.218567 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:11.219128 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:11.219141 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:11.220895 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
W20251025 08:03:11.221153 5598 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:11.216509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:60951 (local address 127.27.132.62:46479)
1025 08:03:11.216682 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:11.216686 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:11.216746 (+ 60us) server_negotiation.cc:408] Connection header received
1025 08:03:11.216847 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:11.216850 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:11.216896 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:11.216963 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:11.217478 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.218015 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:11.218555 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.218678 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:11.221037 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:60951: BlockingRecv error: recv got EOF from 127.27.132.2:60951 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Oct 25 08:03:11 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:11 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:11.401327 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:11.396259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:58415)
1025 08:03:11.396527 (+ 268us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:11.396544 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:11.396657 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:11.396935 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:11.396938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:11.397149 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:11.397435 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:11.397443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.398283 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:11.398287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:11.398800 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:11.398808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.398911 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:11.399523 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:11.399535 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:11.401219 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
W20251025 08:03:11.401453 5598 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:11.396454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:58415 (local address 127.27.132.62:46479)
1025 08:03:11.396611 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:11.396615 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:11.396631 (+ 16us) server_negotiation.cc:408] Connection header received
1025 08:03:11.396771 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:11.396774 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:11.396830 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:11.396899 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:11.397561 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.398181 (+ 620us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:11.398913 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:11.399112 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:11.401349 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:58415: BlockingRecv error: recv got EOF from 127.27.132.3:58415 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Oct 25 08:03:12 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:12 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:12.034435 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:12.029525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:42275)
1025 08:03:12.029791 (+ 266us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:12.029804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:12.029899 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:12.030142 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:12.030144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:12.030333 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:12.030550 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:12.030556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.031252 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:12.031254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:12.031686 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:12.031694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.031784 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:12.032414 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:12.032428 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:12.034321 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
W20251025 08:03:12.034592 5602 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:12.029598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:42275 (local address 127.27.132.62:46479)
1025 08:03:12.029855 (+ 257us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:12.029859 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:12.029872 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:03:12.029983 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:12.029986 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:12.030037 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:12.030117 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:12.030654 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.031168 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:12.031811 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.032107 (+ 296us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:12.034442 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:42275: BlockingRecv error: recv got EOF from 127.27.132.1:42275 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
Oct 25 08:03:12 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:12 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:12.226468 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:12.221765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:39957)
1025 08:03:12.222029 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:12.222044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:12.222150 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:12.222391 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:12.222394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:12.222592 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:12.222789 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:12.222795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.223524 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:12.223527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:12.223914 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:12.223921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.224013 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:12.224579 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:12.224593 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:12.226351 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
W20251025 08:03:12.226583 5602 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:12.221838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:39957 (local address 127.27.132.62:46479)
1025 08:03:12.221999 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:12.222003 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:12.222107 (+ 104us) server_negotiation.cc:408] Connection header received
1025 08:03:12.222226 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:12.222228 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:12.222274 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:12.222343 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:12.222882 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.223426 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:12.224000 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.224124 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:12.226476 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:39957: BlockingRecv error: recv got EOF from 127.27.132.2:39957 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Oct 25 08:03:12 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:12 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:12.406916 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:12.402202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:49793)
1025 08:03:12.402474 (+ 272us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:12.402486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:12.402607 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:12.402860 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:12.402863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:12.403062 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:12.403248 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:12.403254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.403998 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:12.404004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:12.404416 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:12.404427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.404545 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:12.405113 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:12.405126 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:12.406790 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
W20251025 08:03:12.406991 5602 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:12.402253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:49793 (local address 127.27.132.62:46479)
1025 08:03:12.402394 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:12.402399 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:12.402551 (+ 152us) server_negotiation.cc:408] Connection header received
1025 08:03:12.402706 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:12.402709 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:12.402756 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:12.402830 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:12.403387 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.403890 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:12.404528 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:12.404726 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:12.406901 (+ 2175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:49793: BlockingRecv error: recv got EOF from 127.27.132.3:49793 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Oct 25 08:03:13 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:13 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:13.040436 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:13.035355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:39435)
1025 08:03:13.035639 (+ 284us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:13.035653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:13.035770 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:13.036178 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:13.036183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:13.036427 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:13.036640 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:13.036647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.037592 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:13.037596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:13.037998 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:13.038006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.038101 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:13.038646 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:13.038658 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:13.040324 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
W20251025 08:03:13.040568 5606 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:13.035446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:39435 (local address 127.27.132.62:46479)
1025 08:03:13.035884 (+ 438us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:13.035889 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:13.035904 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:03:13.035951 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:13.035954 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:13.036009 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:13.036100 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:13.036757 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.037428 (+ 671us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:13.038108 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.038245 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:13.040453 (+ 2208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:39435: BlockingRecv error: recv got EOF from 127.27.132.1:39435 (error 108)
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":87,"threads_started":1}
Oct 25 08:03:13 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:13 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:13.232218 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:13.227301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:47727)
1025 08:03:13.227644 (+ 343us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:13.227658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:13.227774 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:13.228038 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:13.228041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:13.228239 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:13.228424 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:13.228430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.229170 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:13.229174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:13.229553 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:13.229560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.229665 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:13.230254 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:13.230266 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:13.232056 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":99,"threads_started":1}
W20251025 08:03:13.232301 5606 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:13.227422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:47727 (local address 127.27.132.62:46479)
1025 08:03:13.227570 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:13.227574 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:13.227737 (+ 163us) server_negotiation.cc:408] Connection header received
1025 08:03:13.227878 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:13.227881 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:13.227925 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:13.227996 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:13.228553 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.229058 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:13.229662 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.229900 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:13.232187 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:47727: BlockingRecv error: recv got EOF from 127.27.132.2:47727 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Oct 25 08:03:13 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:13 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:13.412477 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:13.407706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:34513)
1025 08:03:13.407998 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:13.408014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:13.408144 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:13.408404 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:13.408407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:13.408603 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:13.408814 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:13.408821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.409485 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:13.409488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:13.409876 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:13.409883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.409979 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:13.410513 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:13.410524 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:13.412316 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
W20251025 08:03:13.412586 5606 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:13.407858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:34513 (local address 127.27.132.62:46479)
1025 08:03:13.408000 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:13.408006 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:13.408085 (+ 79us) server_negotiation.cc:408] Connection header received
1025 08:03:13.408222 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:13.408224 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:13.408276 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:13.408341 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:13.408911 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.409388 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:13.409979 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:13.410111 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:13.412449 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:34513: BlockingRecv error: recv got EOF from 127.27.132.3:34513 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Oct 25 08:03:14 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:14 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:14.049264 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:14.041229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:60515)
1025 08:03:14.041505 (+ 276us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:14.041518 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:14.041615 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:14.041885 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:14.041887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:14.042064 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:14.042254 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:14.042260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.043166 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:14.043176 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:14.043662 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:14.043671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.043766 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:14.047413 (+ 3647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:14.047427 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:14.049151 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
W20251025 08:03:14.049471 5610 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:14.041294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:60515 (local address 127.27.132.62:46479)
1025 08:03:14.041563 (+ 269us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:14.041568 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:14.041583 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:03:14.041699 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:14.041702 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:14.041763 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:14.041862 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:14.042396 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.043055 (+ 659us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:14.043759 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.043894 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:14.049305 (+ 5411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:60515: BlockingRecv error: recv got EOF from 127.27.132.1:60515 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
Oct 25 08:03:14 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:14 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:14.238467 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:14.233166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:38751)
1025 08:03:14.233469 (+ 303us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:14.233483 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:14.233583 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:14.233890 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:14.233893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:14.234072 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:14.234263 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:14.234269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.235115 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:14.235120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:14.235575 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:14.235583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.235684 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:14.236293 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:14.236307 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:14.238352 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":90,"threads_started":1}
W20251025 08:03:14.238655 5610 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:14.233377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:38751 (local address 127.27.132.62:46479)
1025 08:03:14.233515 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:14.233519 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:14.233535 (+ 16us) server_negotiation.cc:408] Connection header received
1025 08:03:14.233715 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:14.233719 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:14.233773 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:14.233853 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:14.234395 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.234935 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:14.235711 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.235870 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:14.238520 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:38751: BlockingRecv error: recv got EOF from 127.27.132.2:38751 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Oct 25 08:03:14 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:14 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:14.417901 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:14.413292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:49807)
1025 08:03:14.413582 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:14.413595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:14.413711 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:14.413961 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:14.413963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:14.414146 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:14.414337 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:14.414342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.415058 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:14.415060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:14.415472 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:14.415478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.415597 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:14.416137 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:14.416149 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:14.417788 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":70,"threads_started":1}
W20251025 08:03:14.418030 5610 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:14.413403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:49807 (local address 127.27.132.62:46479)
1025 08:03:14.413593 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:14.413596 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:14.413694 (+ 98us) server_negotiation.cc:408] Connection header received
1025 08:03:14.413787 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:14.413790 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:14.413841 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:14.413926 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:14.414473 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.414968 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:14.415606 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:14.415739 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:14.417924 (+ 2185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:49807: BlockingRecv error: recv got EOF from 127.27.132.3:49807 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Oct 25 08:03:15 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:15 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:15.057688 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:15.050012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:51275)
1025 08:03:15.050279 (+ 267us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:15.050292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:15.050382 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:15.050627 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:15.050629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:15.050811 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:15.051002 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:15.051008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.051843 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:15.051846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:15.052218 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:15.052225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.052314 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:15.055687 (+ 3373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:15.055702 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:15.057564 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
W20251025 08:03:15.057885 5614 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:15.050099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:51275 (local address 127.27.132.62:46479)
1025 08:03:15.050369 (+ 270us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:15.050373 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:15.050386 (+ 13us) server_negotiation.cc:408] Connection header received
1025 08:03:15.050452 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:15.050455 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:15.050525 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:15.050602 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:15.051145 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.051740 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:15.052304 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.052431 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:15.057739 (+ 5308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:51275: BlockingRecv error: recv got EOF from 127.27.132.1:51275 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":71,"threads_started":1}
Oct 25 08:03:15 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:15 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:15.244222 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:15.239359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:49711)
1025 08:03:15.239765 (+ 406us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:15.239780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:15.239919 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:15.240200 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:15.240205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:15.240445 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:15.240710 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:15.240721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.241416 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:15.241419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:15.241804 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:15.241810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.241892 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:15.242411 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:15.242423 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:15.244113 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":103,"threads_started":1}
W20251025 08:03:15.244340 5614 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:15.239487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:49711 (local address 127.27.132.62:46479)
1025 08:03:15.239649 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:15.239655 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:15.239880 (+ 225us) server_negotiation.cc:408] Connection header received
1025 08:03:15.240021 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:15.240026 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:15.240082 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:15.240162 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:15.240829 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.241327 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:15.241890 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.242017 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:15.244239 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:49711: BlockingRecv error: recv got EOF from 127.27.132.2:49711 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Oct 25 08:03:15 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:15 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.3@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:15.423427 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:15.418642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.3:49193)
1025 08:03:15.418904 (+ 262us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:15.418917 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:15.419023 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:15.419367 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:15.419373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:15.419574 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:15.419826 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:15.419834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.420579 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:15.420583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:15.420960 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:15.420967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.421059 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:15.421571 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:15.421584 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:15.423270 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
W20251025 08:03:15.423548 5614 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:15.418905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:49193 (local address 127.27.132.62:46479)
1025 08:03:15.419059 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:15.419066 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:15.419092 (+ 26us) server_negotiation.cc:408] Connection header received
1025 08:03:15.419129 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:15.419134 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:15.419192 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:15.419275 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:15.419951 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.420470 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:15.421047 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:15.421168 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:15.423440 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.3:49193: BlockingRecv error: recv got EOF from 127.27.132.3:49193 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Oct 25 08:03:16 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:16 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.1@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:16.064718 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:16.058599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.1:49607)
1025 08:03:16.058881 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:16.058898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:16.059010 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:16.059296 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:16.059301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:16.059568 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:16.059864 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:16.059875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.060985 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:16.060989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:16.061434 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:16.061443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.061564 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:16.062193 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:16.062213 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:16.064556 (+ 2343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
W20251025 08:03:16.064816 5618 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:16.058755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:49607 (local address 127.27.132.62:46479)
1025 08:03:16.059053 (+ 298us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:16.059057 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:16.059072 (+ 15us) server_negotiation.cc:408] Connection header received
1025 08:03:16.059115 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:16.059118 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:16.059161 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:16.059263 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:16.060000 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.060723 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:16.061553 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.061757 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:16.064686 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.1:49607: BlockingRecv error: recv got EOF from 127.27.132.1:49607 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":72,"threads_started":1}
Oct 25 08:03:16 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
Oct 25 08:03:16 dist-test-slave-lq8l krb5kdc[4551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.132.2@KRBTEST.COM for oryx/127.27.132.62@KRBTEST.COM, Server not found in Kerberos database
I20251025 08:03:16.250068 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:16.245187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:46479 (local address 127.27.132.2:40457)
1025 08:03:16.245478 (+ 291us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:16.245491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:16.245589 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:16.245820 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:16.245822 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:16.246003 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:16.246195 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:16.246200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.246915 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:16.246918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:16.247401 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:16.247412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.247523 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:16.248195 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:16.248208 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:16.249957 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.132.62:46479: Server oryx/127.27.132.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":82,"threads_started":1}
W20251025 08:03:16.250211 5618 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:16.245290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:40457 (local address 127.27.132.62:46479)
1025 08:03:16.245473 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:16.245480 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:16.245555 (+ 75us) server_negotiation.cc:408] Connection header received
1025 08:03:16.245661 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:16.245663 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:16.245705 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:16.245776 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:16.246290 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.246776 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:16.247511 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.247720 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:16.250079 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.132.2:40457: BlockingRecv error: recv got EOF from 127.27.132.2:40457 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20251025 08:03:16.327459 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 5143
I20251025 08:03:16.333263 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 5295
I20251025 08:03:16.338790 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 5427
I20251025 08:03:16.344205 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 5073
2025-10-25T08:03:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18925 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-10-25T08:03:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:03:16Z Disabled control of system clock
I20251025 08:03:16.373548 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:42429
--webserver_interface=127.27.132.62
--webserver_port=0
--builtin_ntp_servers=127.27.132.20:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:42429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251025 08:03:16.479933 5623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:16.480132 5623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:16.480171 5623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:16.483230 5623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:03:16.483281 5623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:16.483299 5623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:16.483316 5623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:03:16.483355 5623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:03:16.487027 5623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:42429
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:42429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:16.488109 5623 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:16.489164 5623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:16.495123 5631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:16.495123 5628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:16.495159 5623 server_base.cc:1047] running on GCE node
W20251025 08:03:16.495198 5629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:16.495862 5623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:16.496752 5623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:16.497912 5623 hybrid_clock.cc:648] HybridClock initialized: now 1761379396497891 us; error 44 us; skew 500 ppm
I20251025 08:03:16.499670 5623 webserver.cc:492] Webserver started at http://127.27.132.62:36421/ using document root <none> and password file <none>
I20251025 08:03:16.500241 5623 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:16.500303 5623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:16.500483 5623 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:16.502189 5623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "a08af490bfa94c8dbdc729906e26c64c"
format_stamp: "Formatted at 2025-10-25 08:03:16 on dist-test-slave-lq8l"
server_key: "7c81d86bb84e2d8c807f7254592f87ff"
server_key_iv: "006c4eb2cfefdc37d29ee72294d5e3a6"
server_key_version: "encryptionkey@0"
I20251025 08:03:16.502674 5623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "a08af490bfa94c8dbdc729906e26c64c"
format_stamp: "Formatted at 2025-10-25 08:03:16 on dist-test-slave-lq8l"
server_key: "7c81d86bb84e2d8c807f7254592f87ff"
server_key_iv: "006c4eb2cfefdc37d29ee72294d5e3a6"
server_key_version: "encryptionkey@0"
I20251025 08:03:16.506101 5623 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251025 08:03:16.508282 5637 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:16.509315 5623 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:03:16.509429 5623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "a08af490bfa94c8dbdc729906e26c64c"
format_stamp: "Formatted at 2025-10-25 08:03:16 on dist-test-slave-lq8l"
server_key: "7c81d86bb84e2d8c807f7254592f87ff"
server_key_iv: "006c4eb2cfefdc37d29ee72294d5e3a6"
server_key_version: "encryptionkey@0"
I20251025 08:03:16.509526 5623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:16.520576 5623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:16.521170 5623 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:16.521332 5623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:16.528283 5623 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:42429
I20251025 08:03:16.528280 5689 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:42429 every 8 connection(s)
I20251025 08:03:16.529265 5623 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:03:16.532027 5690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:03:16.537369 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c: Bootstrap starting.
I20251025 08:03:16.538564 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 5623
I20251025 08:03:16.538700 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:03:16.538969 28176 external_mini_cluster.cc:1468] Setting key 56abf241926407a6aa55587e7305add5
I20251025 08:03:16.539950 5690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c: Neither blocks nor log segments found. Creating new log.
I20251025 08:03:16.540721 5690 log.cc:826] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c: Log is configured to *not* fsync() on all Append() calls
I20251025 08:03:16.542706 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c: No bootstrap required, opened a new log
I20251025 08:03:16.545629 5690 raft_consensus.cc:359] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08af490bfa94c8dbdc729906e26c64c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42429 } }
I20251025 08:03:16.545836 5690 raft_consensus.cc:385] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:03:16.545904 5690 raft_consensus.cc:740] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a08af490bfa94c8dbdc729906e26c64c, State: Initialized, Role: FOLLOWER
I20251025 08:03:16.546300 5690 consensus_queue.cc:260] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08af490bfa94c8dbdc729906e26c64c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42429 } }
I20251025 08:03:16.546420 5690 raft_consensus.cc:399] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:03:16.546468 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:16.540075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.27.132.62:42429)
1025 08:03:16.540675 (+ 600us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:16.540684 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:16.540717 (+ 33us) server_negotiation.cc:408] Connection header received
1025 08:03:16.541376 (+ 659us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:16.541403 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:16.541733 (+ 330us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:16.542002 (+ 269us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:16.542521 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.543281 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:16.544120 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.544303 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:16.544926 (+ 623us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:16.544943 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:16.544953 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:03:16.545315 (+ 362us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:16.545351 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:03:16.545365 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:16.545443 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:16.545604 (+ 161us) server_negotiation.cc:300] Negotiation successful
1025 08:03:16.545838 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20251025 08:03:16.546484 5690 raft_consensus.cc:493] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:03:16.546649 5690 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:03:16.547449 5690 raft_consensus.cc:515] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08af490bfa94c8dbdc729906e26c64c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42429 } }
I20251025 08:03:16.547806 5690 leader_election.cc:304] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a08af490bfa94c8dbdc729906e26c64c; no voters:
I20251025 08:03:16.548094 5690 leader_election.cc:290] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:03:16.548182 5695 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:03:16.548393 5695 raft_consensus.cc:697] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [term 1 LEADER]: Becoming Leader. State: Replica: a08af490bfa94c8dbdc729906e26c64c, State: Running, Role: LEADER
I20251025 08:03:16.548727 5695 consensus_queue.cc:237] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08af490bfa94c8dbdc729906e26c64c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42429 } }
I20251025 08:03:16.549145 5690 sys_catalog.cc:565] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:03:16.550132 5697 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a08af490bfa94c8dbdc729906e26c64c. Latest consensus state: current_term: 1 leader_uuid: "a08af490bfa94c8dbdc729906e26c64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08af490bfa94c8dbdc729906e26c64c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42429 } } }
I20251025 08:03:16.550233 5697 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [sys.catalog]: This master's current role is: LEADER
I20251025 08:03:16.550560 5696 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a08af490bfa94c8dbdc729906e26c64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08af490bfa94c8dbdc729906e26c64c" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42429 } } }
I20251025 08:03:16.550647 5696 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c [sys.catalog]: This master's current role is: LEADER
I20251025 08:03:16.550799 5704 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:03:16.553718 5704 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:03:16.558647 5704 catalog_manager.cc:1357] Generated new cluster ID: 3526c1519e1b456dbbd383d9deb759a7
I20251025 08:03:16.558714 5704 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:03:16.588263 5704 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:03:16.589193 5704 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:03:16.596516 5704 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a08af490bfa94c8dbdc729906e26c64c: Generated new TSK 0
I20251025 08:03:16.597231 5704 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 08:03:16.599601 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:42429
--builtin_ntp_servers=127.27.132.20:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251025 08:03:16.711922 5714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:16.712141 5714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:16.712188 5714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:16.715297 5714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:16.715376 5714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:16.715493 5714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:03:16.719128 5714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:42429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:16.720217 5714 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:16.721318 5714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:16.727597 5719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:16.727639 5720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:16.727676 5722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:16.727900 5714 server_base.cc:1047] running on GCE node
I20251025 08:03:16.728259 5714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:16.728858 5714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:16.730026 5714 hybrid_clock.cc:648] HybridClock initialized: now 1761379396729992 us; error 45 us; skew 500 ppm
I20251025 08:03:16.732121 5714 webserver.cc:492] Webserver started at http://127.27.132.1:40373/ using document root <none> and password file <none>
I20251025 08:03:16.732837 5714 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:16.732911 5714 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:16.733134 5714 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:16.735224 5714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "eb74fab7b8a846db8582726285da5f98"
format_stamp: "Formatted at 2025-10-25 08:03:16 on dist-test-slave-lq8l"
server_key: "efaab22b95496b84f77a8f010e558828"
server_key_iv: "bba6fbc24d0d10ffbcdd79905273ae80"
server_key_version: "encryptionkey@0"
I20251025 08:03:16.735769 5714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "eb74fab7b8a846db8582726285da5f98"
format_stamp: "Formatted at 2025-10-25 08:03:16 on dist-test-slave-lq8l"
server_key: "efaab22b95496b84f77a8f010e558828"
server_key_iv: "bba6fbc24d0d10ffbcdd79905273ae80"
server_key_version: "encryptionkey@0"
I20251025 08:03:16.739377 5714 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:03:16.741453 5728 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:16.742412 5714 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:03:16.742522 5714 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "eb74fab7b8a846db8582726285da5f98"
format_stamp: "Formatted at 2025-10-25 08:03:16 on dist-test-slave-lq8l"
server_key: "efaab22b95496b84f77a8f010e558828"
server_key_iv: "bba6fbc24d0d10ffbcdd79905273ae80"
server_key_version: "encryptionkey@0"
I20251025 08:03:16.742623 5714 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:16.757553 5714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:16.758255 5714 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:16.758447 5714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:16.759016 5714 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:16.760020 5714 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:16.760071 5714 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:16.760131 5714 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:16.760164 5714 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:16.769222 5714 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:44949
I20251025 08:03:16.769275 5841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:44949 every 8 connection(s)
I20251025 08:03:16.770198 5714 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:03:16.775947 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 5714
I20251025 08:03:16.776077 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:03:16.776362 28176 external_mini_cluster.cc:1468] Setting key c5809801bf6341aedd50a52b247fa202
I20251025 08:03:16.778323 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:16.772048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:58477 (local address 127.27.132.62:42429)
1025 08:03:16.772232 (+ 184us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:16.772237 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:16.773194 (+ 957us) server_negotiation.cc:408] Connection header received
1025 08:03:16.773957 (+ 763us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:16.773960 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:16.774011 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:16.774084 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:16.775020 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.775582 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:16.776232 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.776433 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:16.777779 (+ 1346us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:16.777798 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:16.777801 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:03:16.777814 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:16.777837 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:03:16.777848 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:16.777938 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:16.778069 (+ 131us) server_negotiation.cc:300] Negotiation successful
1025 08:03:16.778145 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20251025 08:03:16.778373 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:42429
--builtin_ntp_servers=127.27.132.20:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251025 08:03:16.778959 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:16.772374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42429 (local address 127.27.132.1:58477)
1025 08:03:16.773052 (+ 678us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:16.773085 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:16.773790 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:16.774230 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:16.774237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:16.774313 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:16.774886 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:16.774897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.775685 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:16.775688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:16.776110 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:16.776119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.776309 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:16.777517 (+ 1208us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:03:16.777522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:16.777577 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:03:16.777648 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:16.777939 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:16.777944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:16.777948 (+ 4us) client_negotiation.cc:769] Sending connection context
1025 08:03:16.778056 (+ 108us) client_negotiation.cc:240] Negotiation successful
1025 08:03:16.778244 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
I20251025 08:03:16.780041 5842 heartbeater.cc:344] Connected to a master server at 127.27.132.62:42429
I20251025 08:03:16.780275 5842 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:16.780814 5842 heartbeater.cc:507] Master 127.27.132.62:42429 requested a full tablet report, sending...
I20251025 08:03:16.782223 5654 ts_manager.cc:194] Registered new tserver with Master: eb74fab7b8a846db8582726285da5f98 (127.27.132.1:44949)
I20251025 08:03:16.783423 5654 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.27.132.1:58477
W20251025 08:03:16.888410 5845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:16.888638 5845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:16.888711 5845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:16.891953 5845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:16.892015 5845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:16.892127 5845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:03:16.895978 5845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:42429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:16.897094 5845 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:16.898204 5845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:16.904603 5851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:16.904594 5850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:16.904696 5845 server_base.cc:1047] running on GCE node
W20251025 08:03:16.904595 5853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:16.905121 5845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:16.905668 5845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:16.906821 5845 hybrid_clock.cc:648] HybridClock initialized: now 1761379396906815 us; error 33 us; skew 500 ppm
I20251025 08:03:16.908555 5845 webserver.cc:492] Webserver started at http://127.27.132.2:34269/ using document root <none> and password file <none>
I20251025 08:03:16.909104 5845 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:16.909165 5845 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:16.909341 5845 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:16.911015 5845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "d3ca0aad595643cdade8f7a470e3bdd7"
format_stamp: "Formatted at 2025-10-25 08:03:16 on dist-test-slave-lq8l"
server_key: "4462595af820c966fc23c7cd9ea41ab5"
server_key_iv: "6c9fb026d5e1407360241c617c9c3975"
server_key_version: "encryptionkey@0"
I20251025 08:03:16.911522 5845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "d3ca0aad595643cdade8f7a470e3bdd7"
format_stamp: "Formatted at 2025-10-25 08:03:16 on dist-test-slave-lq8l"
server_key: "4462595af820c966fc23c7cd9ea41ab5"
server_key_iv: "6c9fb026d5e1407360241c617c9c3975"
server_key_version: "encryptionkey@0"
I20251025 08:03:16.914825 5845 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20251025 08:03:16.917047 5859 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:16.918030 5845 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:03:16.918143 5845 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "d3ca0aad595643cdade8f7a470e3bdd7"
format_stamp: "Formatted at 2025-10-25 08:03:16 on dist-test-slave-lq8l"
server_key: "4462595af820c966fc23c7cd9ea41ab5"
server_key_iv: "6c9fb026d5e1407360241c617c9c3975"
server_key_version: "encryptionkey@0"
I20251025 08:03:16.918237 5845 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:16.940589 5845 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:16.941282 5845 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:16.941458 5845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:16.942023 5845 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:16.942987 5845 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:16.943037 5845 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:16.943104 5845 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:16.943130 5845 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:16.952473 5845 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:40991
I20251025 08:03:16.952533 5972 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:40991 every 8 connection(s)
I20251025 08:03:16.953464 5845 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:03:16.954388 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 5845
I20251025 08:03:16.954488 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:03:16.954722 28176 external_mini_cluster.cc:1468] Setting key 6e487370d20ae34cd609ede7b48e309f
I20251025 08:03:16.956705 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:42429
--builtin_ntp_servers=127.27.132.20:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251025 08:03:16.961889 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:16.955410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:57597 (local address 127.27.132.62:42429)
1025 08:03:16.955562 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:16.955567 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:16.956561 (+ 994us) server_negotiation.cc:408] Connection header received
1025 08:03:16.957461 (+ 900us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:16.957464 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:16.957528 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:16.957608 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:16.958595 (+ 987us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.959094 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:16.959802 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.959984 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:16.961367 (+ 1383us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:16.961387 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:16.961389 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:03:16.961402 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:16.961422 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:03:16.961429 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:16.961496 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:16.961663 (+ 167us) server_negotiation.cc:300] Negotiation successful
1025 08:03:16.961730 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251025 08:03:16.962564 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:16.955708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42429 (local address 127.27.132.2:57597)
1025 08:03:16.956412 (+ 704us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:16.956455 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:16.957264 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:16.957782 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:16.957792 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:16.957876 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:16.958447 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:16.958459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.959215 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:16.959218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:16.959668 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:16.959676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:16.959900 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:16.961137 (+ 1237us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:03:16.961139 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:16.961172 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:03:16.961228 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:16.961509 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:16.961521 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:16.961525 (+ 4us) client_negotiation.cc:769] Sending connection context
1025 08:03:16.961665 (+ 140us) client_negotiation.cc:240] Negotiation successful
1025 08:03:16.961909 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20251025 08:03:16.963837 5973 heartbeater.cc:344] Connected to a master server at 127.27.132.62:42429
I20251025 08:03:16.964074 5973 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:16.964622 5973 heartbeater.cc:507] Master 127.27.132.62:42429 requested a full tablet report, sending...
I20251025 08:03:16.965665 5654 ts_manager.cc:194] Registered new tserver with Master: d3ca0aad595643cdade8f7a470e3bdd7 (127.27.132.2:40991)
I20251025 08:03:16.966218 5654 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.27.132.2:57597
W20251025 08:03:17.066000 5976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:17.066242 5976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:17.066320 5976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:17.069473 5976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:17.069545 5976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:17.069648 5976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:03:17.073367 5976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:42429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:17.074437 5976 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:17.075646 5976 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:17.082100 5982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:17.082109 5984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:17.082109 5981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:17.082343 5976 server_base.cc:1047] running on GCE node
I20251025 08:03:17.082872 5976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:17.083421 5976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:17.084575 5976 hybrid_clock.cc:648] HybridClock initialized: now 1761379397084566 us; error 24 us; skew 500 ppm
I20251025 08:03:17.086315 5976 webserver.cc:492] Webserver started at http://127.27.132.3:34173/ using document root <none> and password file <none>
I20251025 08:03:17.086853 5976 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:17.086906 5976 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:17.087085 5976 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:17.088768 5976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "d6f4748f290a420997fcec4bd1713ad1"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "79bc90de239d8d95ff9e246565e52ae0"
server_key_iv: "6f630c539b97675d3bf962b0aaeceb28"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.089218 5976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "d6f4748f290a420997fcec4bd1713ad1"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "79bc90de239d8d95ff9e246565e52ae0"
server_key_iv: "6f630c539b97675d3bf962b0aaeceb28"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.092530 5976 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:03:17.094636 5990 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.095644 5976 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:03:17.095746 5976 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "d6f4748f290a420997fcec4bd1713ad1"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "79bc90de239d8d95ff9e246565e52ae0"
server_key_iv: "6f630c539b97675d3bf962b0aaeceb28"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.095844 5976 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:17.106979 5976 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:17.107645 5976 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:17.107815 5976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:17.108315 5976 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:17.109248 5976 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:17.109298 5976 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.109360 5976 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:17.109390 5976 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.118590 5976 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:34835
I20251025 08:03:17.118625 6103 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:34835 every 8 connection(s)
I20251025 08:03:17.119632 5976 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:03:17.122493 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 5976
I20251025 08:03:17.122596 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:03:17.122830 28176 external_mini_cluster.cc:1468] Setting key 5396baf409b7a7bfd5b40e4f4fcf00ca
I20251025 08:03:17.127810 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:17.121379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:38421 (local address 127.27.132.62:42429)
1025 08:03:17.121524 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:17.121528 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:17.122427 (+ 899us) server_negotiation.cc:408] Connection header received
1025 08:03:17.123393 (+ 966us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:17.123397 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:17.123451 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:17.123554 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:17.124808 (+ 1254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:17.125380 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:17.125990 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:17.126121 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:17.127392 (+ 1271us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:17.127407 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:17.127409 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:03:17.127421 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:17.127438 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:03:17.127446 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:17.127507 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:17.127655 (+ 148us) server_negotiation.cc:300] Negotiation successful
1025 08:03:17.127703 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251025 08:03:17.128437 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:17.121660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42429 (local address 127.27.132.3:38421)
1025 08:03:17.122289 (+ 629us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:17.122323 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:17.123135 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:17.123774 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:17.123787 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:17.123898 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:17.124661 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:17.124679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:17.125493 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:17.125496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:17.125896 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:17.125903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:17.126076 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:17.127165 (+ 1089us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:03:17.127167 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:17.127198 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:03:17.127249 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:17.127516 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:17.127522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:17.127525 (+ 3us) client_negotiation.cc:769] Sending connection context
1025 08:03:17.127646 (+ 121us) client_negotiation.cc:240] Negotiation successful
1025 08:03:17.127837 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20251025 08:03:17.129371 6104 heartbeater.cc:344] Connected to a master server at 127.27.132.62:42429
I20251025 08:03:17.129572 6104 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:17.130057 6104 heartbeater.cc:507] Master 127.27.132.62:42429 requested a full tablet report, sending...
I20251025 08:03:17.130935 5654 ts_manager.cc:194] Registered new tserver with Master: d6f4748f290a420997fcec4bd1713ad1 (127.27.132.3:34835)
I20251025 08:03:17.131521 5654 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.27.132.3:38421
I20251025 08:03:17.136693 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251025 08:03:17.139206 5693 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:17.138481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.27.132.62:42429)
1025 08:03:17.138616 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:17.138620 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:17.138719 (+ 99us) server_negotiation.cc:408] Connection header received
1025 08:03:17.138820 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:17.138823 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:17.138864 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:17.138931 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:17.139088 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47180: BlockingRecv error: recv got EOF from 127.0.0.1:47180 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20251025 08:03:17.139645 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 5714
I20251025 08:03:17.147888 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 5845
I20251025 08:03:17.155103 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 5976
I20251025 08:03:17.162338 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 5623
2025-10-25T08:03:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (818 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-10-25T08:03:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:03:17Z Disabled control of system clock
I20251025 08:03:17.191843 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:40475
--webserver_interface=127.27.132.62
--webserver_port=0
--builtin_ntp_servers=127.27.132.20:43759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:40475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251025 08:03:17.299244 6115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:17.299571 6115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:17.299636 6115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:17.302973 6115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:03:17.303045 6115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:17.303061 6115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:17.303079 6115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:03:17.303100 6115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:03:17.307123 6115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:40475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:40475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:17.308293 6115 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:17.309433 6115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:17.315052 6120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:17.314992 6123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:17.314991 6121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:17.315246 6115 server_base.cc:1047] running on GCE node
I20251025 08:03:17.315801 6115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:17.316695 6115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:17.317847 6115 hybrid_clock.cc:648] HybridClock initialized: now 1761379397317838 us; error 32 us; skew 500 ppm
I20251025 08:03:17.319530 6115 webserver.cc:492] Webserver started at http://127.27.132.62:40147/ using document root <none> and password file <none>
I20251025 08:03:17.320092 6115 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:17.320154 6115 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:17.320335 6115 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:17.321997 6115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "71cdfd8057ff472a9ed671e431d733a3"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "273c13854cd3802d02e212b59910fdba"
server_key_iv: "e2fefdcd6b67ba49db49e7504187a10f"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.322451 6115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "71cdfd8057ff472a9ed671e431d733a3"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "273c13854cd3802d02e212b59910fdba"
server_key_iv: "e2fefdcd6b67ba49db49e7504187a10f"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.325690 6115 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20251025 08:03:17.327818 6129 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.328791 6115 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20251025 08:03:17.328910 6115 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "71cdfd8057ff472a9ed671e431d733a3"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "273c13854cd3802d02e212b59910fdba"
server_key_iv: "e2fefdcd6b67ba49db49e7504187a10f"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.328995 6115 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:17.333997 6115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:17.334542 6115 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:17.334704 6115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:17.341452 6181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:40475 every 8 connection(s)
I20251025 08:03:17.341454 6115 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:40475
I20251025 08:03:17.342478 6115 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:03:17.345207 6182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:03:17.346923 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 6115
I20251025 08:03:17.347069 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:03:17.347366 28176 external_mini_cluster.cc:1468] Setting key 0d1639af66f9aa0728c8389fb33ad790
I20251025 08:03:17.351696 6182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3: Bootstrap starting.
I20251025 08:03:17.352689 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:17.348852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.27.132.62:40475)
1025 08:03:17.349484 (+ 632us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:17.349493 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:17.349526 (+ 33us) server_negotiation.cc:408] Connection header received
1025 08:03:17.350044 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:17.350061 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:17.350359 (+ 298us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:17.350673 (+ 314us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:17.351619 (+ 946us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:17.351624 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:17.351633 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:03:17.351643 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:17.351686 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:03:17.351697 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:17.351844 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:17.351934 (+ 90us) server_negotiation.cc:300] Negotiation successful
1025 08:03:17.352143 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":131,"threads_started":1}
I20251025 08:03:17.354017 6182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3: Neither blocks nor log segments found. Creating new log.
I20251025 08:03:17.354879 6182 log.cc:826] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3: Log is configured to *not* fsync() on all Append() calls
I20251025 08:03:17.356911 6182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3: No bootstrap required, opened a new log
I20251025 08:03:17.359299 6182 raft_consensus.cc:359] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71cdfd8057ff472a9ed671e431d733a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40475 } }
I20251025 08:03:17.359546 6182 raft_consensus.cc:385] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:03:17.359601 6182 raft_consensus.cc:740] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71cdfd8057ff472a9ed671e431d733a3, State: Initialized, Role: FOLLOWER
I20251025 08:03:17.359961 6182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71cdfd8057ff472a9ed671e431d733a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40475 } }
I20251025 08:03:17.360069 6182 raft_consensus.cc:399] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:03:17.360112 6182 raft_consensus.cc:493] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:03:17.360176 6182 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:03:17.361016 6182 raft_consensus.cc:515] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71cdfd8057ff472a9ed671e431d733a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40475 } }
I20251025 08:03:17.361323 6182 leader_election.cc:304] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71cdfd8057ff472a9ed671e431d733a3; no voters:
I20251025 08:03:17.361585 6182 leader_election.cc:290] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:03:17.361723 6187 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:03:17.361927 6187 raft_consensus.cc:697] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [term 1 LEADER]: Becoming Leader. State: Replica: 71cdfd8057ff472a9ed671e431d733a3, State: Running, Role: LEADER
I20251025 08:03:17.362219 6187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71cdfd8057ff472a9ed671e431d733a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40475 } }
I20251025 08:03:17.362708 6182 sys_catalog.cc:565] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:03:17.363763 6189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71cdfd8057ff472a9ed671e431d733a3. Latest consensus state: current_term: 1 leader_uuid: "71cdfd8057ff472a9ed671e431d733a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71cdfd8057ff472a9ed671e431d733a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40475 } } }
I20251025 08:03:17.363790 6188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71cdfd8057ff472a9ed671e431d733a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71cdfd8057ff472a9ed671e431d733a3" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 40475 } } }
I20251025 08:03:17.363914 6189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [sys.catalog]: This master's current role is: LEADER
I20251025 08:03:17.363914 6188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3 [sys.catalog]: This master's current role is: LEADER
I20251025 08:03:17.364537 6196 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:03:17.367197 6196 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:03:17.372364 6196 catalog_manager.cc:1357] Generated new cluster ID: 1e74c34f48a047528d00b5c3e32a8ae8
I20251025 08:03:17.372447 6196 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:03:17.386569 6196 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:03:17.386891 6196 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:03:17.394833 6196 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 71cdfd8057ff472a9ed671e431d733a3: Generated new TSK 0
I20251025 08:03:17.395648 6196 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 08:03:17.402388 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:40475
--builtin_ntp_servers=127.27.132.20:43759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251025 08:03:17.509966 6206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:17.510195 6206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:17.510248 6206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:17.513538 6206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:17.513595 6206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:17.513715 6206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:03:17.517779 6206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:17.518821 6206 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:17.519968 6206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:17.526234 6211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:17.526252 6214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:17.526333 6212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:17.526459 6206 server_base.cc:1047] running on GCE node
I20251025 08:03:17.526978 6206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:17.527643 6206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:17.528795 6206 hybrid_clock.cc:648] HybridClock initialized: now 1761379397528783 us; error 38 us; skew 500 ppm
I20251025 08:03:17.530578 6206 webserver.cc:492] Webserver started at http://127.27.132.1:42207/ using document root <none> and password file <none>
I20251025 08:03:17.531240 6206 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:17.531317 6206 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:17.531584 6206 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:17.533313 6206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "b665211c285a41e9a905b50366e48c2f"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "388b4fe5480eeb7a3868e9513a676339"
server_key_iv: "90e7046778d2bb96265e6658e8f7acac"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.533753 6206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "b665211c285a41e9a905b50366e48c2f"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "388b4fe5480eeb7a3868e9513a676339"
server_key_iv: "90e7046778d2bb96265e6658e8f7acac"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.537091 6206 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20251025 08:03:17.539209 6220 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.540242 6206 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:03:17.540354 6206 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "b665211c285a41e9a905b50366e48c2f"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "388b4fe5480eeb7a3868e9513a676339"
server_key_iv: "90e7046778d2bb96265e6658e8f7acac"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.540450 6206 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:17.545562 6206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:17.546128 6206 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:17.546289 6206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:17.546802 6206 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:17.547771 6206 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:17.547823 6206 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.547883 6206 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:17.547914 6206 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.556710 6206 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:43407
I20251025 08:03:17.556758 6333 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:43407 every 8 connection(s)
I20251025 08:03:17.557627 6206 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:03:17.558007 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 6206
I20251025 08:03:17.558142 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:03:17.558425 28176 external_mini_cluster.cc:1468] Setting key 12a165cf6224c1501242c37b104d4913
I20251025 08:03:17.560371 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:40475
--builtin_ntp_servers=127.27.132.20:43759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20251025 08:03:17.563086 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:17.559297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:39261 (local address 127.27.132.62:40475)
1025 08:03:17.559457 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:17.559462 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:17.560487 (+ 1025us) server_negotiation.cc:408] Connection header received
1025 08:03:17.561286 (+ 799us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:17.561292 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:17.561346 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:17.561445 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:17.562666 (+ 1221us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:17.562670 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:17.562672 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:03:17.562682 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:17.562701 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:03:17.562709 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:17.562783 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:17.562946 (+ 163us) server_negotiation.cc:300] Negotiation successful
1025 08:03:17.562981 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251025 08:03:17.564184 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:17.559657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:40475 (local address 127.27.132.1:39261)
1025 08:03:17.560332 (+ 675us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:17.560375 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:17.561101 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:17.561647 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:17.561661 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:17.561774 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:17.562399 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:03:17.562403 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:17.562451 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:03:17.562519 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:17.562787 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:17.562797 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:17.562811 (+ 14us) client_negotiation.cc:769] Sending connection context
1025 08:03:17.563011 (+ 200us) client_negotiation.cc:240] Negotiation successful
1025 08:03:17.563310 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":101,"threads_started":1}
I20251025 08:03:17.565601 6334 heartbeater.cc:344] Connected to a master server at 127.27.132.62:40475
I20251025 08:03:17.565879 6334 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:17.566385 6334 heartbeater.cc:507] Master 127.27.132.62:40475 requested a full tablet report, sending...
I20251025 08:03:17.568127 6146 ts_manager.cc:194] Registered new tserver with Master: b665211c285a41e9a905b50366e48c2f (127.27.132.1:43407)
W20251025 08:03:17.668339 6337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:17.668545 6337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:17.668601 6337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:17.671736 6337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:17.671787 6337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:17.671859 6337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:03:17.675447 6337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:17.676479 6337 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:17.677557 6337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:17.683724 6342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:17.683789 6345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:17.683724 6343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:17.683969 6337 server_base.cc:1047] running on GCE node
I20251025 08:03:17.684345 6337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:17.684870 6337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:17.686015 6337 hybrid_clock.cc:648] HybridClock initialized: now 1761379397685990 us; error 38 us; skew 500 ppm
I20251025 08:03:17.687780 6337 webserver.cc:492] Webserver started at http://127.27.132.2:35417/ using document root <none> and password file <none>
I20251025 08:03:17.688318 6337 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:17.688373 6337 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:17.688561 6337 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:17.690207 6337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "658b26ac87564886a9cd322e726b6399"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "dfdaf4e383da916b5a836ffbf01713ee"
server_key_iv: "5b0d01485134089d366da50d971c3782"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.690690 6337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "658b26ac87564886a9cd322e726b6399"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "dfdaf4e383da916b5a836ffbf01713ee"
server_key_iv: "5b0d01485134089d366da50d971c3782"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.694105 6337 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:03:17.696422 6351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.697435 6337 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:03:17.697561 6337 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "658b26ac87564886a9cd322e726b6399"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "dfdaf4e383da916b5a836ffbf01713ee"
server_key_iv: "5b0d01485134089d366da50d971c3782"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.697651 6337 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:17.702358 6337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:17.702893 6337 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:17.703042 6337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:17.703576 6337 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:17.704473 6337 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:17.704528 6337 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.704586 6337 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:17.704613 6337 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.713234 6337 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:33167
I20251025 08:03:17.713284 6464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:33167 every 8 connection(s)
I20251025 08:03:17.714159 6337 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:03:17.716246 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 6337
I20251025 08:03:17.716356 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:03:17.716632 28176 external_mini_cluster.cc:1468] Setting key f5f0dec9a9f0bb4170a945d1da3d39c4
I20251025 08:03:17.718456 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:40475
--builtin_ntp_servers=127.27.132.20:43759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20251025 08:03:17.719672 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:17.716027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:51857 (local address 127.27.132.62:40475)
1025 08:03:17.716200 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:17.716206 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:17.717198 (+ 992us) server_negotiation.cc:408] Connection header received
1025 08:03:17.717969 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:17.717973 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:17.718014 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:17.718090 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:17.719154 (+ 1064us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:17.719157 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:17.719160 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:03:17.719172 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:17.719196 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:03:17.719204 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:17.719284 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:17.719495 (+ 211us) server_negotiation.cc:300] Negotiation successful
1025 08:03:17.719540 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251025 08:03:17.720592 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:17.716304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:40475 (local address 127.27.132.2:51857)
1025 08:03:17.717061 (+ 757us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:17.717096 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:17.717785 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:17.718230 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:17.718238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:17.718317 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:17.718934 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:03:17.718936 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:17.718965 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:03:17.719015 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:17.719282 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:17.719290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:17.719297 (+ 7us) client_negotiation.cc:769] Sending connection context
1025 08:03:17.719555 (+ 258us) client_negotiation.cc:240] Negotiation successful
1025 08:03:17.719924 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":107,"threads_started":1}
I20251025 08:03:17.721908 6465 heartbeater.cc:344] Connected to a master server at 127.27.132.62:40475
I20251025 08:03:17.722174 6465 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:17.722687 6465 heartbeater.cc:507] Master 127.27.132.62:40475 requested a full tablet report, sending...
I20251025 08:03:17.723832 6146 ts_manager.cc:194] Registered new tserver with Master: 658b26ac87564886a9cd322e726b6399 (127.27.132.2:33167)
W20251025 08:03:17.825490 6468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:17.825717 6468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:17.825768 6468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:17.829049 6468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:17.829102 6468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:17.829176 6468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:03:17.832998 6468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:43759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:17.834131 6468 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:17.835242 6468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:17.841663 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:17.841665 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:17.841684 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:17.842077 6468 server_base.cc:1047] running on GCE node
I20251025 08:03:17.842422 6468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:17.842952 6468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:17.844098 6468 hybrid_clock.cc:648] HybridClock initialized: now 1761379397844071 us; error 42 us; skew 500 ppm
I20251025 08:03:17.845876 6468 webserver.cc:492] Webserver started at http://127.27.132.3:43261/ using document root <none> and password file <none>
I20251025 08:03:17.846464 6468 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:17.846526 6468 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:17.846714 6468 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:17.848412 6468 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "e39de71b6ff64673810653f460d21a5e"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "a83c9a17a7bfa4e94149257c1bd8cfff"
server_key_iv: "b2508acb794cc1e92e0622438c6ac932"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.848920 6468 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "e39de71b6ff64673810653f460d21a5e"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "a83c9a17a7bfa4e94149257c1bd8cfff"
server_key_iv: "b2508acb794cc1e92e0622438c6ac932"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.852301 6468 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251025 08:03:17.854429 6482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.855468 6468 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251025 08:03:17.855571 6468 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "e39de71b6ff64673810653f460d21a5e"
format_stamp: "Formatted at 2025-10-25 08:03:17 on dist-test-slave-lq8l"
server_key: "a83c9a17a7bfa4e94149257c1bd8cfff"
server_key_iv: "b2508acb794cc1e92e0622438c6ac932"
server_key_version: "encryptionkey@0"
I20251025 08:03:17.855659 6468 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:17.860359 6468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:17.860909 6468 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:17.861070 6468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:17.861557 6468 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:17.862461 6468 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:17.862511 6468 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.862566 6468 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:17.862602 6468 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:17.871517 6468 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:44597
I20251025 08:03:17.871564 6595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:44597 every 8 connection(s)
I20251025 08:03:17.872478 6468 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
I20251025 08:03:17.873816 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 6468
I20251025 08:03:17.873948 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:03:17.874228 28176 external_mini_cluster.cc:1468] Setting key 8216b03d8d958ec36b630f5631f2e5d5
I20251025 08:03:17.877894 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:17.874204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:33645 (local address 127.27.132.62:40475)
1025 08:03:17.874348 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:17.874353 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:17.875363 (+ 1010us) server_negotiation.cc:408] Connection header received
1025 08:03:17.876275 (+ 912us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:17.876279 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:17.876320 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:17.876395 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:17.877435 (+ 1040us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:17.877440 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:17.877442 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1025 08:03:17.877454 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:17.877476 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1025 08:03:17.877486 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:17.877564 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:17.877715 (+ 151us) server_negotiation.cc:300] Negotiation successful
1025 08:03:17.877758 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251025 08:03:17.878548 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:17.874478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:40475 (local address 127.27.132.3:33645)
1025 08:03:17.875215 (+ 737us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:17.875250 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:17.876074 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:17.876543 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:17.876557 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:17.876639 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:17.877220 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1025 08:03:17.877221 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:17.877251 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1025 08:03:17.877303 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:17.877566 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:17.877577 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:17.877583 (+ 6us) client_negotiation.cc:769] Sending connection context
1025 08:03:17.877769 (+ 186us) client_negotiation.cc:240] Negotiation successful
1025 08:03:17.877978 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":152,"threads_started":1}
I20251025 08:03:17.879685 6596 heartbeater.cc:344] Connected to a master server at 127.27.132.62:40475
I20251025 08:03:17.879899 6596 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:17.880265 6596 heartbeater.cc:507] Master 127.27.132.62:40475 requested a full tablet report, sending...
I20251025 08:03:17.881188 6146 ts_manager.cc:194] Registered new tserver with Master: e39de71b6ff64673810653f460d21a5e (127.27.132.3:44597)
I20251025 08:03:17.887861 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251025 08:03:17.890334 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 08:03:17.889654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.27.132.62:40475)
1025 08:03:17.889771 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:17.889774 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:17.889880 (+ 106us) server_negotiation.cc:408] Connection header received
1025 08:03:17.889975 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:17.889977 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:17.890010 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:17.890084 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:17.890239 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50862: BlockingRecv error: recv got EOF from 127.0.0.1:50862 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20251025 08:03:17.890647 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 6206
I20251025 08:03:17.896724 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 6337
I20251025 08:03:17.902547 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 6468
I20251025 08:03:17.907750 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 6115
2025-10-25T08:03:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (744 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:03:17 dist-test-slave-lq8l krb5kdc[6606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:03:17 dist-test-slave-lq8l krb5kdc[6606](info): set up 2 sockets
Oct 25 08:03:17 dist-test-slave-lq8l krb5kdc[6606](info): commencing operation
krb5kdc: starting...
W20251025 08:03:19.897009 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.968s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:03:19 dist-test-slave-lq8l krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:03:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:03:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:03:20.032250 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:34793
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:34793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:03:20.145984 6622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:20.146339 6622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:20.146442 6622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:20.149995 6622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:03:20.150074 6622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:20.150099 6622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:20.150117 6622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:03:20.150134 6622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:03:20.154620 6622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:34793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:34793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:20.155782 6622 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:20.156646 6622 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:20.162480 6627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:20.162555 6628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:20.162495 6630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:20.162825 6622 server_base.cc:1047] running on GCE node
I20251025 08:03:20.163308 6622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:20.164247 6622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:20.165395 6622 hybrid_clock.cc:648] HybridClock initialized: now 1761379400165378 us; error 41 us; skew 500 ppm
Oct 25 08:03:20 dist-test-slave-lq8l krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:03:20.168135 6622 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:03:20.169201 6622 webserver.cc:492] Webserver started at http://127.27.132.62:42937/ using document root <none> and password file <none>
I20251025 08:03:20.169752 6622 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:20.169816 6622 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:20.170020 6622 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:20.171790 6622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "b59097b941734754afee82748e2687c2"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "b9923437b8fd4e5c5d9ad1f9e1a5c48c"
server_key_iv: "ec7f3a04293a00a150cb3fc7eba847d7"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.172281 6622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "b59097b941734754afee82748e2687c2"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "b9923437b8fd4e5c5d9ad1f9e1a5c48c"
server_key_iv: "ec7f3a04293a00a150cb3fc7eba847d7"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.175668 6622 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251025 08:03:20.177870 6637 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:20.178845 6622 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:03:20.178954 6622 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "b59097b941734754afee82748e2687c2"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "b9923437b8fd4e5c5d9ad1f9e1a5c48c"
server_key_iv: "ec7f3a04293a00a150cb3fc7eba847d7"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.179042 6622 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:20.197250 6622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:20.204630 6622 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:20.204835 6622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:20.212133 6622 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:34793
I20251025 08:03:20.212155 6689 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:34793 every 8 connection(s)
I20251025 08:03:20.213131 6622 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:03:20.216025 6690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:03:20.218322 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 6622
I20251025 08:03:20.218497 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:03:20.218853 28176 external_mini_cluster.cc:1468] Setting key 93b81e1d92d7647677b0fbd3cb8feea6
I20251025 08:03:20.222734 6690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2: Bootstrap starting.
I20251025 08:03:20.225047 6690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2: Neither blocks nor log segments found. Creating new log.
I20251025 08:03:20.225836 6690 log.cc:826] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2: Log is configured to *not* fsync() on all Append() calls
Oct 25 08:03:20 dist-test-slave-lq8l krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:03:20.227882 6690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2: No bootstrap required, opened a new log
I20251025 08:03:20.230212 6690 raft_consensus.cc:359] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59097b941734754afee82748e2687c2" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34793 } }
I20251025 08:03:20.230394 6690 raft_consensus.cc:385] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:03:20.230456 6690 raft_consensus.cc:740] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59097b941734754afee82748e2687c2, State: Initialized, Role: FOLLOWER
I20251025 08:03:20.230921 6690 consensus_queue.cc:260] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59097b941734754afee82748e2687c2" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34793 } }
I20251025 08:03:20.231052 6690 raft_consensus.cc:399] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:03:20.231117 6690 raft_consensus.cc:493] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:03:20.231210 6690 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:03:20.231278 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.220255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.27.132.62:34793)
1025 08:03:20.220674 (+ 419us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:20.220684 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:20.220718 (+ 34us) server_negotiation.cc:408] Connection header received
1025 08:03:20.221235 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:20.221251 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:20.221526 (+ 275us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:20.221822 (+ 296us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:20.222726 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.223554 (+ 828us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.224183 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.224462 (+ 279us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.226947 (+ 2485us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:20.226965 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:20.226977 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:03:20.227013 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:20.228896 (+ 1883us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:20.229293 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:20.229298 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:20.229302 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:20.229370 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:20.229592 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:20.229596 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:20.229600 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:20.229911 (+ 311us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:20.230119 (+ 208us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:20.230381 (+ 262us) server_negotiation.cc:300] Negotiation successful
1025 08:03:20.230646 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
I20251025 08:03:20.232056 6690 raft_consensus.cc:515] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59097b941734754afee82748e2687c2" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34793 } }
I20251025 08:03:20.232381 6690 leader_election.cc:304] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b59097b941734754afee82748e2687c2; no voters:
I20251025 08:03:20.232678 6690 leader_election.cc:290] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:03:20.232791 6695 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:03:20.232975 6695 raft_consensus.cc:697] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [term 1 LEADER]: Becoming Leader. State: Replica: b59097b941734754afee82748e2687c2, State: Running, Role: LEADER
I20251025 08:03:20.233255 6695 consensus_queue.cc:237] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59097b941734754afee82748e2687c2" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34793 } }
I20251025 08:03:20.233803 6690 sys_catalog.cc:565] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:03:20.234396 6697 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b59097b941734754afee82748e2687c2. Latest consensus state: current_term: 1 leader_uuid: "b59097b941734754afee82748e2687c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59097b941734754afee82748e2687c2" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34793 } } }
I20251025 08:03:20.234499 6697 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [sys.catalog]: This master's current role is: LEADER
I20251025 08:03:20.234746 6696 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b59097b941734754afee82748e2687c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59097b941734754afee82748e2687c2" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 34793 } } }
I20251025 08:03:20.234846 6696 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2 [sys.catalog]: This master's current role is: LEADER
I20251025 08:03:20.235080 6699 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:03:20.238070 6699 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:03:20.243571 6699 catalog_manager.cc:1357] Generated new cluster ID: 5cbf569d722c45e88a7fe7bb7a052c53
I20251025 08:03:20.243638 6699 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:03:20.261089 6699 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:03:20.262010 6699 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:03:20.269433 6699 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2: Generated new TSK 0
I20251025 08:03:20.270108 6699 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:03:20.323823 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34793
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:03:20.431169 6718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:20.431447 6718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:20.431522 6718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:20.434758 6718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:20.434835 6718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:20.434935 6718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:03:20.439297 6718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:20.440416 6718 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:20.441272 6718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251025 08:03:20.447943 6718 server_base.cc:1047] running on GCE node
W20251025 08:03:20.447880 6726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:20.447887 6723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:20.447880 6724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:20.448462 6718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:20.449030 6718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:20.450201 6718 hybrid_clock.cc:648] HybridClock initialized: now 1761379400450183 us; error 30 us; skew 500 ppm
Oct 25 08:03:20 dist-test-slave-lq8l krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:03:20.453094 6718 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:03:20.454130 6718 webserver.cc:492] Webserver started at http://127.27.132.1:38101/ using document root <none> and password file <none>
I20251025 08:03:20.454684 6718 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:20.454738 6718 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:20.454936 6718 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:20.456950 6718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "5dc5cc85205b473ca52a67be731def98"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "b9f9c8dfcfbf5339c40290eba02e655f"
server_key_iv: "584be543f0ba94e751c18523d7c893c0"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.457553 6718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "5dc5cc85205b473ca52a67be731def98"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "b9f9c8dfcfbf5339c40290eba02e655f"
server_key_iv: "584be543f0ba94e751c18523d7c893c0"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.461863 6718 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20251025 08:03:20.464057 6733 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:20.465097 6718 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:03:20.465200 6718 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "5dc5cc85205b473ca52a67be731def98"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "b9f9c8dfcfbf5339c40290eba02e655f"
server_key_iv: "584be543f0ba94e751c18523d7c893c0"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.465305 6718 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:20.482407 6718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:20.485360 6718 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:20.485530 6718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:20.486088 6718 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:20.486964 6718 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:20.487013 6718 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:20.487074 6718 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:20.487105 6718 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:20.496186 6718 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:32833
I20251025 08:03:20.496208 6846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:32833 every 8 connection(s)
I20251025 08:03:20.497189 6718 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:03:20.499734 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 6718
I20251025 08:03:20.499884 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:03:20.500175 28176 external_mini_cluster.cc:1468] Setting key 93d3e2f5e5957913ee28bac18a044f75
Oct 25 08:03:20 dist-test-slave-lq8l krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
I20251025 08:03:20.511032 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.498981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:33701 (local address 127.27.132.62:34793)
1025 08:03:20.499113 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:20.499117 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:20.499861 (+ 744us) server_negotiation.cc:408] Connection header received
1025 08:03:20.500927 (+ 1066us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:20.500931 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:20.500994 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:20.501075 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:20.502708 (+ 1633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.503429 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.504089 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.504234 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.507268 (+ 3034us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:20.507314 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:20.507318 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:03:20.507378 (+ 60us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:20.508993 (+ 1615us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:20.509493 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:20.509497 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:20.509498 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:20.509544 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:20.509816 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:20.509819 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:20.509822 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:20.510008 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:20.510103 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:20.510657 (+ 554us) server_negotiation.cc:300] Negotiation successful
1025 08:03:20.510801 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251025 08:03:20.511742 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.499193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34793 (local address 127.27.132.1:33701)
1025 08:03:20.499703 (+ 510us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:20.499754 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:20.500721 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:20.501244 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:20.501252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:20.501746 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:20.502523 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.502545 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.503569 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.503572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:20.503977 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.503984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.504169 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.505311 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:20.505337 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:20.507081 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:20.509123 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:20.509130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:20.509142 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:20.509386 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:20.509633 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:20.509638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:20.509640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:20.509728 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:20.510117 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:20.510127 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:20.510377 (+ 250us) client_negotiation.cc:769] Sending connection context
1025 08:03:20.510612 (+ 235us) client_negotiation.cc:240] Negotiation successful
1025 08:03:20.510912 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
I20251025 08:03:20.513021 6847 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34793
I20251025 08:03:20.513276 6847 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:20.513823 6847 heartbeater.cc:507] Master 127.27.132.62:34793 requested a full tablet report, sending...
I20251025 08:03:20.515296 6654 ts_manager.cc:194] Registered new tserver with Master: 5dc5cc85205b473ca52a67be731def98 (127.27.132.1:32833)
I20251025 08:03:20.516480 6654 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:33701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:03:20.547530 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34793
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:03:20.653450 6854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:20.653689 6854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:20.653749 6854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:20.656956 6854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:20.657019 6854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:20.657112 6854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:03:20.661414 6854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:20.662456 6854 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:20.663304 6854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:20.669574 6860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:20.669601 6862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:20.669875 6854 server_base.cc:1047] running on GCE node
W20251025 08:03:20.669603 6859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:20.670228 6854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:20.670734 6854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:20.671880 6854 hybrid_clock.cc:648] HybridClock initialized: now 1761379400671852 us; error 41 us; skew 500 ppm
Oct 25 08:03:20 dist-test-slave-lq8l krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:03:20.674558 6854 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:03:20.675603 6854 webserver.cc:492] Webserver started at http://127.27.132.2:41155/ using document root <none> and password file <none>
I20251025 08:03:20.676143 6854 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:20.676198 6854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:20.676376 6854 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:20.678004 6854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "6b028d3997004b5a83f1f81be8b55328"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "1c2042d9dd08e4afe32b65d3e3f061a1"
server_key_iv: "600f2e1048b5aecca8f7273a3e6ffa46"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.678454 6854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "6b028d3997004b5a83f1f81be8b55328"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "1c2042d9dd08e4afe32b65d3e3f061a1"
server_key_iv: "600f2e1048b5aecca8f7273a3e6ffa46"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.681764 6854 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:03:20.683921 6869 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:20.684898 6854 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:03:20.685005 6854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "6b028d3997004b5a83f1f81be8b55328"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "1c2042d9dd08e4afe32b65d3e3f061a1"
server_key_iv: "600f2e1048b5aecca8f7273a3e6ffa46"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.685091 6854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:20.696321 6854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:20.699122 6854 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:20.699295 6854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:20.699826 6854 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:20.700608 6854 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:20.700657 6854 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:20.700712 6854 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:20.700747 6854 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:20.710134 6854 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:42987
I20251025 08:03:20.710239 6982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:42987 every 8 connection(s)
I20251025 08:03:20.711095 6854 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:03:20.712953 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 6854
I20251025 08:03:20.713074 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:03:20.713325 28176 external_mini_cluster.cc:1468] Setting key 360a68f3f722ce85c9014ff9c9da4b8b
Oct 25 08:03:20 dist-test-slave-lq8l krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.132.3@KRBTEST.COM; defaulting to no policy
I20251025 08:03:20.724144 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.712888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.2:60359 (local address 127.27.132.62:34793)
1025 08:03:20.713070 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:20.713075 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:20.713717 (+ 642us) server_negotiation.cc:408] Connection header received
1025 08:03:20.714610 (+ 893us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:20.714613 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:20.714665 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:20.714740 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:20.716479 (+ 1739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.717340 (+ 861us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.717965 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.718198 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.720660 (+ 2462us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:20.720682 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:20.720688 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:03:20.720717 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:20.722395 (+ 1678us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:20.722897 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:20.722901 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:20.722902 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:20.722951 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:20.723170 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:20.723172 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:20.723173 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:20.723307 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:20.723418 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:20.723920 (+ 502us) server_negotiation.cc:300] Negotiation successful
1025 08:03:20.724026 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251025 08:03:20.724838 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.713151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34793 (local address 127.27.132.2:60359)
1025 08:03:20.713575 (+ 424us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:20.713617 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:20.714423 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:20.714914 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:20.714924 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:20.715526 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:20.716289 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.716309 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.717457 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.717460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:20.717850 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.717857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.718022 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.719029 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:20.719050 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:20.720496 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:20.722518 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:20.722522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:20.722531 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:20.722791 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:20.723030 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:20.723032 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:20.723034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:20.723104 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:20.723428 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:20.723438 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:20.723681 (+ 243us) client_negotiation.cc:769] Sending connection context
1025 08:03:20.723848 (+ 167us) client_negotiation.cc:240] Negotiation successful
1025 08:03:20.724050 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.3@KRBTEST.COM" created.
I20251025 08:03:20.725909 6983 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34793
I20251025 08:03:20.726186 6983 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:20.726809 6983 heartbeater.cc:507] Master 127.27.132.62:34793 requested a full tablet report, sending...
I20251025 08:03:20.727825 6654 ts_manager.cc:194] Registered new tserver with Master: 6b028d3997004b5a83f1f81be8b55328 (127.27.132.2:42987)
I20251025 08:03:20.728360 6654 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.2@KRBTEST.COM'} at 127.27.132.2:60359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:03:20.760455 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.3:0
--local_ip_for_outbound_sockets=127.27.132.3
--webserver_interface=127.27.132.3
--webserver_port=0
--tserver_master_addrs=127.27.132.62:34793
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:03:20.865835 6990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:03:20.866058 6990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:03:20.866132 6990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:03:20.869400 6990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:03:20.869462 6990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:03:20.869540 6990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.3
I20251025 08:03:20.873646 6990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.132.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.3
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:03:20.874599 6990 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:03:20.875494 6990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:03:20.881651 6995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:20.881628 6998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:03:20.881821 6996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:03:20.882025 6990 server_base.cc:1047] running on GCE node
I20251025 08:03:20.882408 6990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:03:20.883010 6990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:03:20.884173 6990 hybrid_clock.cc:648] HybridClock initialized: now 1761379400884146 us; error 49 us; skew 500 ppm
Oct 25 08:03:20 dist-test-slave-lq8l krb5kdc[6606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:03:20.886989 6990 init.cc:377] Logged in from keytab as kudu/127.27.132.3@KRBTEST.COM (short username kudu)
I20251025 08:03:20.888077 6990 webserver.cc:492] Webserver started at http://127.27.132.3:41777/ using document root <none> and password file <none>
I20251025 08:03:20.888603 6990 fs_manager.cc:362] Metadata directory not provided
I20251025 08:03:20.888675 6990 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:03:20.888875 6990 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 08:03:20.890511 6990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/instance:
uuid: "2e1ffb469a2b4485832276d965124ba9"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "02cd79aa24de7cd3a15f6d98f7bab136"
server_key_iv: "323408d828d05f2ce5dac10330eaf6fc"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.890967 6990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance:
uuid: "2e1ffb469a2b4485832276d965124ba9"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "02cd79aa24de7cd3a15f6d98f7bab136"
server_key_iv: "323408d828d05f2ce5dac10330eaf6fc"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.894272 6990 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251025 08:03:20.896442 7005 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:20.897449 6990 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251025 08:03:20.897569 6990 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
uuid: "2e1ffb469a2b4485832276d965124ba9"
format_stamp: "Formatted at 2025-10-25 08:03:20 on dist-test-slave-lq8l"
server_key: "02cd79aa24de7cd3a15f6d98f7bab136"
server_key_iv: "323408d828d05f2ce5dac10330eaf6fc"
server_key_version: "encryptionkey@0"
I20251025 08:03:20.897670 6990 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:03:20.921109 6990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:03:20.923863 6990 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:03:20.924067 6990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:03:20.924623 6990 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:03:20.925462 6990 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:03:20.925509 6990 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:20.925568 6990 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:03:20.925606 6990 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:03:20.934962 6990 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.3:46119
I20251025 08:03:20.934988 7118 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.3:46119 every 8 connection(s)
I20251025 08:03:20.936062 6990 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/data/info.pb
Oct 25 08:03:20 dist-test-slave-lq8l krb5kdc[6606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.3@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:03:20.946313 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 6990
I20251025 08:03:20.946424 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal/instance
I20251025 08:03:20.946659 28176 external_mini_cluster.cc:1468] Setting key 28e753800ef456f98b7547b2dd909b1c
I20251025 08:03:20.948114 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.937762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.3:51043 (local address 127.27.132.62:34793)
1025 08:03:20.937905 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:20.937908 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:20.938537 (+ 629us) server_negotiation.cc:408] Connection header received
1025 08:03:20.939381 (+ 844us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:20.939384 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:20.939441 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:20.939521 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:20.940745 (+ 1224us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.941232 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.941839 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.942013 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.944673 (+ 2660us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:20.944690 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:20.944693 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:03:20.944715 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:20.946008 (+ 1293us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:20.946517 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:20.946520 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:20.946522 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:20.946562 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:20.946853 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:20.946856 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:20.946859 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:20.947040 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:20.947134 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:20.947839 (+ 705us) server_negotiation.cc:300] Negotiation successful
1025 08:03:20.947989 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251025 08:03:20.948791 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.937993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:34793 (local address 127.27.132.3:51043)
1025 08:03:20.938392 (+ 399us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:20.938427 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:20.939160 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:20.939687 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:20.939695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:20.940069 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:03:20.940600 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.940611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.941346 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.941349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:20.941730 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.941737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.941940 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.943100 (+ 1160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:03:20.943122 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:03:20.944531 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:03:20.946138 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:20.946143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:20.946154 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:20.946390 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:20.946648 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:03:20.946651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:03:20.946653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:03:20.946764 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:03:20.947158 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:03:20.947165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:03:20.947468 (+ 303us) client_negotiation.cc:769] Sending connection context
1025 08:03:20.947711 (+ 243us) client_negotiation.cc:240] Negotiation successful
1025 08:03:20.947996 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20251025 08:03:20.949946 7119 heartbeater.cc:344] Connected to a master server at 127.27.132.62:34793
I20251025 08:03:20.950182 7119 heartbeater.cc:461] Registering TS with master...
I20251025 08:03:20.950683 7119 heartbeater.cc:507] Master 127.27.132.62:34793 requested a full tablet report, sending...
I20251025 08:03:20.951645 6654 ts_manager.cc:194] Registered new tserver with Master: 2e1ffb469a2b4485832276d965124ba9 (127.27.132.3:46119)
I20251025 08:03:20.952186 6654 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.3@KRBTEST.COM'} at 127.27.132.3:51043
I20251025 08:03:20.960559 28176 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 08:03:20.968256 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.962346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.27.132.62:34793)
1025 08:03:20.962483 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:20.962487 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:20.962593 (+ 106us) server_negotiation.cc:408] Connection header received
1025 08:03:20.962697 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:20.962699 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:20.962738 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:20.962794 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:03:20.963544 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.964051 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.964616 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.964736 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.965777 (+ 1041us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:03:20.965789 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:03:20.965792 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:03:20.965807 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:03:20.967012 (+ 1205us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:20.967359 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:20.967365 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:20.967370 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:20.967415 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:03:20.967594 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:03:20.967596 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:03:20.967597 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:03:20.967754 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:03:20.967864 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:20.968005 (+ 141us) server_negotiation.cc:300] Negotiation successful
1025 08:03:20.968089 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 08:03:20.970587 6654 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 08:03:20.972976 6654 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 08:03:20.985195 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.981313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:32833 (local address 127.0.0.1:59484)
1025 08:03:20.981898 (+ 585us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:20.981911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:20.981985 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:20.982477 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:20.982483 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:20.982503 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:03:20.982782 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.982791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.983988 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.983991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:20.984766 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.984776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.984894 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.984943 (+ 49us) client_negotiation.cc:769] Sending connection context
1025 08:03:20.985017 (+ 74us) client_negotiation.cc:240] Negotiation successful
1025 08:03:20.985090 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":98,"threads_started":1}
I20251025 08:03:20.985714 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.980721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:42987 (local address 127.0.0.1:53854)
1025 08:03:20.981294 (+ 573us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:20.981323 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:20.981457 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:20.982495 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:20.982498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:20.982529 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:03:20.982831 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.982842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.984421 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.984425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:20.985378 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.985390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.985473 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.985488 (+ 15us) client_negotiation.cc:769] Sending connection context
1025 08:03:20.985526 (+ 38us) client_negotiation.cc:240] Negotiation successful
1025 08:03:20.985581 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"spinlock_wait_cycles":2816,"thread_start_us":79,"threads_started":1}
I20251025 08:03:20.985894 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.981601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.27.132.1:32833)
1025 08:03:20.982052 (+ 451us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:20.982059 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:20.982080 (+ 21us) server_negotiation.cc:408] Connection header received
1025 08:03:20.982165 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:20.982170 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:20.982316 (+ 146us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:20.982461 (+ 145us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:03:20.982950 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.983869 (+ 919us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.984918 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.985450 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.985583 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:20.985647 (+ 64us) server_negotiation.cc:300] Negotiation successful
1025 08:03:20.985741 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":83,"threads_started":1}
I20251025 08:03:20.986896 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.980944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:46119 (local address 127.0.0.1:41954)
1025 08:03:20.981650 (+ 706us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:20.981663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:20.981752 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:20.982697 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:20.982700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:20.982715 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:03:20.982960 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.982966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.984421 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.984425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:20.986542 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:20.986551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.986680 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.986695 (+ 15us) client_negotiation.cc:769] Sending connection context
1025 08:03:20.986744 (+ 49us) client_negotiation.cc:240] Negotiation successful
1025 08:03:20.986793 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":86,"threads_started":1}
I20251025 08:03:20.986907 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.980876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.27.132.2:42987)
1025 08:03:20.982019 (+ 1143us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:20.982028 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:20.982052 (+ 24us) server_negotiation.cc:408] Connection header received
1025 08:03:20.982142 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:20.982150 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:20.982332 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:20.982956 (+ 624us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:03:20.983079 (+ 123us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.984287 (+ 1208us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.985963 (+ 1676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.986497 (+ 534us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.986605 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:20.986670 (+ 65us) server_negotiation.cc:300] Negotiation successful
1025 08:03:20.986764 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":126,"threads_started":1}
I20251025 08:03:20.987650 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:20.981472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.27.132.3:46119)
1025 08:03:20.982282 (+ 810us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:20.982287 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:20.982308 (+ 21us) server_negotiation.cc:408] Connection header received
1025 08:03:20.982387 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:20.982408 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:20.982547 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:20.982702 (+ 155us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:03:20.983080 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.984286 (+ 1206us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:20.986707 (+ 2421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:20.987244 (+ 537us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:20.987368 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:20.987431 (+ 63us) server_negotiation.cc:300] Negotiation successful
1025 08:03:20.987513 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":175,"threads_started":1}
I20251025 08:03:20.988921 6781 tablet_service.cc:1505] Processing CreateTablet for tablet 9d9f91456abf4b13bfe7c9b39adfab36 (DEFAULT_TABLE table=test-table [id=c2b4971f0a4f4cf3a28ee483e089d94b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:03:20.989403 6915 tablet_service.cc:1505] Processing CreateTablet for tablet 9d9f91456abf4b13bfe7c9b39adfab36 (DEFAULT_TABLE table=test-table [id=c2b4971f0a4f4cf3a28ee483e089d94b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:03:20.989969 6781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d9f91456abf4b13bfe7c9b39adfab36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:03:20.990195 6915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d9f91456abf4b13bfe7c9b39adfab36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:03:20.990303 7053 tablet_service.cc:1505] Processing CreateTablet for tablet 9d9f91456abf4b13bfe7c9b39adfab36 (DEFAULT_TABLE table=test-table [id=c2b4971f0a4f4cf3a28ee483e089d94b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 08:03:20.991145 7053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d9f91456abf4b13bfe7c9b39adfab36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:03:20.995096 7136 tablet_bootstrap.cc:492] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328: Bootstrap starting.
I20251025 08:03:20.996428 7137 tablet_bootstrap.cc:492] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98: Bootstrap starting.
I20251025 08:03:20.996888 7138 tablet_bootstrap.cc:492] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9: Bootstrap starting.
I20251025 08:03:20.998214 7136 tablet_bootstrap.cc:654] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328: Neither blocks nor log segments found. Creating new log.
I20251025 08:03:20.998214 7137 tablet_bootstrap.cc:654] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98: Neither blocks nor log segments found. Creating new log.
I20251025 08:03:20.998497 7138 tablet_bootstrap.cc:654] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9: Neither blocks nor log segments found. Creating new log.
I20251025 08:03:20.999092 7136 log.cc:826] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328: Log is configured to *not* fsync() on all Append() calls
I20251025 08:03:20.999092 7137 log.cc:826] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98: Log is configured to *not* fsync() on all Append() calls
I20251025 08:03:20.999140 7138 log.cc:826] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9: Log is configured to *not* fsync() on all Append() calls
I20251025 08:03:21.000938 7137 tablet_bootstrap.cc:492] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98: No bootstrap required, opened a new log
I20251025 08:03:21.000938 7136 tablet_bootstrap.cc:492] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328: No bootstrap required, opened a new log
I20251025 08:03:21.000952 7138 tablet_bootstrap.cc:492] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9: No bootstrap required, opened a new log
I20251025 08:03:21.001127 7138 ts_tablet_manager.cc:1403] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251025 08:03:21.001129 7136 ts_tablet_manager.cc:1403] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251025 08:03:21.001129 7137 ts_tablet_manager.cc:1403] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251025 08:03:21.004068 7138 raft_consensus.cc:359] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b028d3997004b5a83f1f81be8b55328" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 42987 } } peers { permanent_uuid: "2e1ffb469a2b4485832276d965124ba9" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46119 } } peers { permanent_uuid: "5dc5cc85205b473ca52a67be731def98" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 32833 } }
I20251025 08:03:21.004269 7138 raft_consensus.cc:385] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:03:21.004325 7138 raft_consensus.cc:740] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1ffb469a2b4485832276d965124ba9, State: Initialized, Role: FOLLOWER
I20251025 08:03:21.004590 7137 raft_consensus.cc:359] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b028d3997004b5a83f1f81be8b55328" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 42987 } } peers { permanent_uuid: "2e1ffb469a2b4485832276d965124ba9" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46119 } } peers { permanent_uuid: "5dc5cc85205b473ca52a67be731def98" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 32833 } }
I20251025 08:03:21.004727 7138 consensus_queue.cc:260] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b028d3997004b5a83f1f81be8b55328" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 42987 } } peers { permanent_uuid: "2e1ffb469a2b4485832276d965124ba9" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46119 } } peers { permanent_uuid: "5dc5cc85205b473ca52a67be731def98" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 32833 } }
I20251025 08:03:21.004810 7137 raft_consensus.cc:385] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:03:21.004858 7137 raft_consensus.cc:740] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc5cc85205b473ca52a67be731def98, State: Initialized, Role: FOLLOWER
I20251025 08:03:21.004745 7136 raft_consensus.cc:359] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b028d3997004b5a83f1f81be8b55328" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 42987 } } peers { permanent_uuid: "2e1ffb469a2b4485832276d965124ba9" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46119 } } peers { permanent_uuid: "5dc5cc85205b473ca52a67be731def98" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 32833 } }
I20251025 08:03:21.004918 7136 raft_consensus.cc:385] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:03:21.004966 7136 raft_consensus.cc:740] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b028d3997004b5a83f1f81be8b55328, State: Initialized, Role: FOLLOWER
I20251025 08:03:21.005255 7137 consensus_queue.cc:260] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b028d3997004b5a83f1f81be8b55328" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 42987 } } peers { permanent_uuid: "2e1ffb469a2b4485832276d965124ba9" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46119 } } peers { permanent_uuid: "5dc5cc85205b473ca52a67be731def98" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 32833 } }
I20251025 08:03:21.005400 7119 heartbeater.cc:499] Master 127.27.132.62:34793 was elected leader, sending a full tablet report...
I20251025 08:03:21.005379 7136 consensus_queue.cc:260] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b028d3997004b5a83f1f81be8b55328" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 42987 } } peers { permanent_uuid: "2e1ffb469a2b4485832276d965124ba9" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46119 } } peers { permanent_uuid: "5dc5cc85205b473ca52a67be731def98" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 32833 } }
I20251025 08:03:21.005636 7138 ts_tablet_manager.cc:1434] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20251025 08:03:21.006098 7137 ts_tablet_manager.cc:1434] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251025 08:03:21.006098 7136 ts_tablet_manager.cc:1434] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20251025 08:03:21.006448 6983 heartbeater.cc:499] Master 127.27.132.62:34793 was elected leader, sending a full tablet report...
I20251025 08:03:21.006554 6847 heartbeater.cc:499] Master 127.27.132.62:34793 was elected leader, sending a full tablet report...
I20251025 08:03:21.023387 7143 raft_consensus.cc:493] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 08:03:21.023545 7143 raft_consensus.cc:515] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b028d3997004b5a83f1f81be8b55328" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 42987 } } peers { permanent_uuid: "2e1ffb469a2b4485832276d965124ba9" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46119 } } peers { permanent_uuid: "5dc5cc85205b473ca52a67be731def98" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 32833 } }
I20251025 08:03:21.024626 7143 leader_election.cc:290] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b028d3997004b5a83f1f81be8b55328 (127.27.132.2:42987), 2e1ffb469a2b4485832276d965124ba9 (127.27.132.3:46119)
I20251025 08:03:21.028403 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:21.024862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.3:46119 (local address 127.27.132.1:54307)
1025 08:03:21.025138 (+ 276us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:21.025150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:21.025323 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:21.025586 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:21.025590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:21.025625 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:03:21.025947 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:21.025955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:21.026962 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:21.026966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:21.027949 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:21.027960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:21.028090 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:21.028112 (+ 22us) client_negotiation.cc:769] Sending connection context
1025 08:03:21.028163 (+ 51us) client_negotiation.cc:240] Negotiation successful
1025 08:03:21.028223 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"spinlock_wait_cycles":237952,"thread_start_us":114,"threads_started":1}
I20251025 08:03:21.028406 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:21.024795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.2:42987 (local address 127.27.132.1:56373)
1025 08:03:21.025057 (+ 262us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:21.025073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1025 08:03:21.025309 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:03:21.025586 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:03:21.025590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:03:21.025624 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1025 08:03:21.025947 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:21.025955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:21.026962 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:03:21.026966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:03:21.027949 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:03:21.027960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:03:21.028091 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:21.028112 (+ 21us) client_negotiation.cc:769] Sending connection context
1025 08:03:21.028164 (+ 52us) client_negotiation.cc:240] Negotiation successful
1025 08:03:21.028223 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"spinlock_wait_cycles":13824,"thread_start_us":96,"threads_started":1}
I20251025 08:03:21.028802 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:21.025135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:54307 (local address 127.27.132.3:46119)
1025 08:03:21.025253 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:21.025257 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:21.025269 (+ 12us) server_negotiation.cc:408] Connection header received
1025 08:03:21.025430 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:21.025434 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:21.025484 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:21.025574 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:03:21.026059 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:21.026847 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:21.028082 (+ 1235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:21.028542 (+ 460us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:21.028589 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:21.028650 (+ 61us) server_negotiation.cc:300] Negotiation successful
1025 08:03:21.028699 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251025 08:03:21.028802 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:21.024925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:56373 (local address 127.27.132.2:42987)
1025 08:03:21.025192 (+ 267us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:21.025197 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:21.025214 (+ 17us) server_negotiation.cc:408] Connection header received
1025 08:03:21.025430 (+ 216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:21.025434 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:21.025484 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:21.025568 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1025 08:03:21.026076 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:21.026848 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:21.028075 (+ 1227us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:21.028540 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:21.028575 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:21.028643 (+ 68us) server_negotiation.cc:300] Negotiation successful
1025 08:03:21.028694 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20251025 08:03:21.029349 7073 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d9f91456abf4b13bfe7c9b39adfab36" candidate_uuid: "5dc5cc85205b473ca52a67be731def98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1ffb469a2b4485832276d965124ba9" is_pre_election: true
I20251025 08:03:21.029348 6937 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d9f91456abf4b13bfe7c9b39adfab36" candidate_uuid: "5dc5cc85205b473ca52a67be731def98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b028d3997004b5a83f1f81be8b55328" is_pre_election: true
I20251025 08:03:21.029603 7073 raft_consensus.cc:2468] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dc5cc85205b473ca52a67be731def98 in term 0.
I20251025 08:03:21.029603 6937 raft_consensus.cc:2468] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dc5cc85205b473ca52a67be731def98 in term 0.
I20251025 08:03:21.030025 6736 leader_election.cc:304] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1ffb469a2b4485832276d965124ba9, 5dc5cc85205b473ca52a67be731def98; no voters:
I20251025 08:03:21.030231 7143 raft_consensus.cc:2804] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 08:03:21.030303 7143 raft_consensus.cc:493] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 08:03:21.030359 7143 raft_consensus.cc:3060] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:03:21.031270 7143 raft_consensus.cc:515] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b028d3997004b5a83f1f81be8b55328" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 42987 } } peers { permanent_uuid: "2e1ffb469a2b4485832276d965124ba9" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46119 } } peers { permanent_uuid: "5dc5cc85205b473ca52a67be731def98" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 32833 } }
I20251025 08:03:21.031658 7143 leader_election.cc:290] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [CANDIDATE]: Term 1 election: Requested vote from peers 6b028d3997004b5a83f1f81be8b55328 (127.27.132.2:42987), 2e1ffb469a2b4485832276d965124ba9 (127.27.132.3:46119)
I20251025 08:03:21.031980 6937 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d9f91456abf4b13bfe7c9b39adfab36" candidate_uuid: "5dc5cc85205b473ca52a67be731def98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b028d3997004b5a83f1f81be8b55328"
I20251025 08:03:21.031981 7073 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d9f91456abf4b13bfe7c9b39adfab36" candidate_uuid: "5dc5cc85205b473ca52a67be731def98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1ffb469a2b4485832276d965124ba9"
I20251025 08:03:21.032095 7073 raft_consensus.cc:3060] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:03:21.032095 6937 raft_consensus.cc:3060] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:03:21.033245 7073 raft_consensus.cc:2468] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dc5cc85205b473ca52a67be731def98 in term 1.
I20251025 08:03:21.033245 6937 raft_consensus.cc:2468] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dc5cc85205b473ca52a67be731def98 in term 1.
I20251025 08:03:21.033542 6734 leader_election.cc:304] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc5cc85205b473ca52a67be731def98, 6b028d3997004b5a83f1f81be8b55328; no voters:
I20251025 08:03:21.033725 7143 raft_consensus.cc:2804] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:03:21.033938 7143 raft_consensus.cc:697] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 1 LEADER]: Becoming Leader. State: Replica: 5dc5cc85205b473ca52a67be731def98, State: Running, Role: LEADER
I20251025 08:03:21.034235 7143 consensus_queue.cc:237] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b028d3997004b5a83f1f81be8b55328" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 42987 } } peers { permanent_uuid: "2e1ffb469a2b4485832276d965124ba9" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46119 } } peers { permanent_uuid: "5dc5cc85205b473ca52a67be731def98" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 32833 } }
I20251025 08:03:21.037096 6654 catalog_manager.cc:5649] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dc5cc85205b473ca52a67be731def98 (127.27.132.1). New cstate: current_term: 1 leader_uuid: "5dc5cc85205b473ca52a67be731def98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b028d3997004b5a83f1f81be8b55328" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 42987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e1ffb469a2b4485832276d965124ba9" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dc5cc85205b473ca52a67be731def98" member_type: VOTER last_known_addr { host: "127.27.132.1" port: 32833 } health_report { overall_health: HEALTHY } } }
I20251025 08:03:21.051280 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:21.048296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.27.132.1:32833)
1025 08:03:21.048432 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1025 08:03:21.048437 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:03:21.048451 (+ 14us) server_negotiation.cc:408] Connection header received
1025 08:03:21.048576 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:03:21.048579 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:03:21.048632 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:03:21.048716 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
1025 08:03:21.049065 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:21.049560 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:03:21.050202 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:03:21.050381 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:03:21.050471 (+ 90us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1025 08:03:21.050899 (+ 428us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1025 08:03:21.051012 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1025 08:03:21.051099 (+ 87us) server_negotiation.cc:300] Negotiation successful
1025 08:03:21.051148 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251025 08:03:21.056586 6937 raft_consensus.cc:1275] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328 [term 1 FOLLOWER]: Refusing update from remote peer 5dc5cc85205b473ca52a67be731def98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:03:21.056581 7073 raft_consensus.cc:1275] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9 [term 1 FOLLOWER]: Refusing update from remote peer 5dc5cc85205b473ca52a67be731def98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 08:03:21.057181 7143 consensus_queue.cc:1048] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b028d3997004b5a83f1f81be8b55328" member_type: VOTER last_known_addr { host: "127.27.132.2" port: 42987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:03:21.057319 7147 consensus_queue.cc:1048] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1ffb469a2b4485832276d965124ba9" member_type: VOTER last_known_addr { host: "127.27.132.3" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 08:03:21.067171 7152 mvcc.cc:204] Tried to move back new op lower bound from 7214610026720473088 to 7214610026638741504. Current Snapshot: MvccSnapshot[applied={T|T < 7214610026720473088}]
I20251025 08:03:21.078063 6653 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53436:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:03:21.078243 6653 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53436:
table { table_name: "test-table" } modify_external_catalogs: true
I20251025 08:03:21.081009 6653 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b59097b941734754afee82748e2687c2: Sending DeleteTablet for 3 replicas of tablet 9d9f91456abf4b13bfe7c9b39adfab36
I20251025 08:03:21.081748 6915 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d9f91456abf4b13bfe7c9b39adfab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:03:21 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:53854
I20251025 08:03:21.081745 7053 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d9f91456abf4b13bfe7c9b39adfab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:03:21 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:41954
I20251025 08:03:21.081913 6781 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d9f91456abf4b13bfe7c9b39adfab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 08:03:21 UTC) from {username='kudu', principal='kudu/127.27.132.62@KRBTEST.COM'} at 127.0.0.1:59484
I20251025 08:03:21.082242 7160 tablet_replica.cc:333] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9: stopping tablet replica
I20251025 08:03:21.082481 7160 raft_consensus.cc:2243] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:03:21.082597 7162 tablet_replica.cc:333] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98: stopping tablet replica
I20251025 08:03:21.082876 7160 raft_consensus.cc:2272] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:03:21.082882 7162 raft_consensus.cc:2243] T 9d9f91456abf4b13bfe7c9b39adfab36 P 5dc5cc85205b473ca52a67be731def98 [term 1 LEADER]: Raft consensus shutting down.
I20251025 08:03:21.083241 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 6718
I20251025 08:03:21.084584 7161 tablet_replica.cc:333] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328: stopping tablet replica
I20251025 08:03:21.084830 7161 raft_consensus.cc:2243] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 08:03:21.085140 7161 raft_consensus.cc:2272] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 08:03:21.085698 7160 ts_tablet_manager.cc:1916] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:03:21.086422 7161 ts_tablet_manager.cc:1916] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 08:03:21.088330 7160 ts_tablet_manager.cc:1929] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:03:21.088418 7160 log.cc:1199] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-2/wal/wals/9d9f91456abf4b13bfe7c9b39adfab36
I20251025 08:03:21.088776 7160 ts_tablet_manager.cc:1950] T 9d9f91456abf4b13bfe7c9b39adfab36 P 2e1ffb469a2b4485832276d965124ba9: Deleting consensus metadata
I20251025 08:03:21.089011 7161 ts_tablet_manager.cc:1929] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 08:03:21.089097 7161 log.cc:1199] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328: Deleting WAL directory at /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761379300881738-28176-0/minicluster-data/ts-1/wal/wals/9d9f91456abf4b13bfe7c9b39adfab36
I20251025 08:03:21.089469 7161 ts_tablet_manager.cc:1950] T 9d9f91456abf4b13bfe7c9b39adfab36 P 6b028d3997004b5a83f1f81be8b55328: Deleting consensus metadata
I20251025 08:03:21.090004 6640 catalog_manager.cc:4985] TS 2e1ffb469a2b4485832276d965124ba9 (127.27.132.3:46119): tablet 9d9f91456abf4b13bfe7c9b39adfab36 (table test-table [id=c2b4971f0a4f4cf3a28ee483e089d94b]) successfully deleted
W20251025 08:03:21.090162 6639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.132.1:32833 (error 108)
I20251025 08:03:21.090417 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 6854
I20251025 08:03:21.090962 6638 catalog_manager.cc:4985] TS 6b028d3997004b5a83f1f81be8b55328 (127.27.132.2:42987): tablet 9d9f91456abf4b13bfe7c9b39adfab36 (table test-table [id=c2b4971f0a4f4cf3a28ee483e089d94b]) successfully deleted
I20251025 08:03:21.092519 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:03:21.092179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.1:32833 (local address 127.0.0.1:59500)
1025 08:03:21.092342 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1025 08:03:21.092416 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.1:32833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251025 08:03:21.092725 6639 catalog_manager.cc:4712] TS 5dc5cc85205b473ca52a67be731def98 (127.27.132.1:32833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d9f91456abf4b13bfe7c9b39adfab36: Network error: Client connection negotiation failed: client connection to 127.27.132.1:32833: connect: Connection refused (error 111)
I20251025 08:03:21.098510 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 6990
I20251025 08:03:21.104565 28176 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu with pid 6622
2025-10-25T08:03:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3200 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 25 08:03:21 dist-test-slave-lq8l krb5kdc[7166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 08:03:21 dist-test-slave-lq8l krb5kdc[7166](info): set up 2 sockets
Oct 25 08:03:21 dist-test-slave-lq8l krb5kdc[7166](info): commencing operation
krb5kdc: starting...
W20251025 08:03:23.088656 28176 mini_kdc.cc:121] Time spent starting KDC: real 1.958s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/test-user.keytab.
Oct 25 08:03:23 dist-test-slave-lq8l krb5kdc[7166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-25T08:03:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T08:03:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.132.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.132.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.132.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/rangeradmin_127.27.132.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.132.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/rangeradmin_127.27.132.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.132.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.132.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.132.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/rangerlookup_127.27.132.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.132.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/rangerlookup_127.27.132.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.132.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/HTTP_127.27.132.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.132.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/HTTP_127.27.132.20@KRBTEST.COM.keytab.
I20251025 08:03:23.239501 28176 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskXgZzmG/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/postgres -l logfile start
2025-10-25 08:03:25.552 UTC [7197] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-25 08:03:25.552 UTC [7197] LOG: listening on IPv4 address "127.27.132.20", port 41335
2025-10-25 08:03:25.556 UTC [7197] LOG: listening on Unix socket "/tmp/.s.PGSQL.41335"
2025-10-25 08:03:25.562 UTC [7202] LOG: database system was shut down at 2025-10-25 08:03:24 UTC
2025-10-25 08:03:25.566 UTC [7197] LOG: database system is ready to accept connections
I20251025 08:03:27.464629 28176 mini_postgres.cc:96] Postgres bound to 41335
2025-10-25 08:03:27.470 UTC [7208] FATAL: database "slave" does not exist
127.27.132.20:41335 - accepting connections
I20251025 08:03:27.471287 28176 mini_ranger.cc:161] Starting Ranger...
I20251025 08:03:27.489446 28176 mini_ranger.cc:84] Created miniranger Postgres user
I20251025 08:03:27.558444 28176 mini_ranger.cc:87] Created ranger Postgres database
I20251025 08:03:27.558537 28176 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-admin
2025-10-25 08:03:27,866 [I] DB FLAVOR :POSTGRES
2025-10-25 08:03:27,866 [I] --------- Verifying Ranger DB connection ---------
2025-10-25 08:03:27,866 [I] Checking connection..
2025-10-25 08:03:27,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-25 08:03:28,182 [I] Checking connection passed.
2025-10-25 08:03:28,183 [I] --------- Verifying version history table ---------
2025-10-25 08:03:28,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-25 08:03:28,482 [I] Table x_db_version_h does not exist in database ranger
2025-10-25 08:03:28,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-25 08:03:28,805 [I] Table x_db_version_h does not exist in database ranger
2025-10-25 08:03:28,805 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-25 08:03:28,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-25 08:03:29.115 UTC [7275] WARNING: there is no transaction in progress
2025-10-25 08:03:29,121 [I] create_dbversion_catalog.sql file imported successfully
2025-10-25 08:03:29,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-25 08:03:29,426 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-25 08:03:29,426 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-25 08:03:29,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-25 08:03:29,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-25 08:03:30,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lq8l.c.gcp-upstream.internal','N') ;"
2025-10-25 08:03:30,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-25 08:03:30,674 [I] Table x_portal_user does not exist in database ranger
2025-10-25 08:03:30,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-25 08:03:30,992 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-25 08:03:30,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-25 08:03:31,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-25 08:03:31,637 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-25 08:03:31,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-25 08:03:32.457 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.467 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.478 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.489 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.500 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.512 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.556 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.561 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.570 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.579 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.588 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.598 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.605 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.609 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.612 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32.616 UTC [7392] WARNING: there is no transaction in progress
2025-10-25 08:03:32,982 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-25 08:03:32,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-25 08:03:33,284 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-25 08:03:33,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-25 08:03:33,595 [I] Table x_portal_user already exists in database 'ranger'
2025-10-25 08:03:33,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-25 08:03:33,924 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-25 08:03:33,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-25 08:03:34,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-25 08:03:34,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lq8l.c.gcp-upstream.internal';"
2025-10-25 08:03:34,834 [I] CORE_DB_SCHEMA import status has been updated
2025-10-25 08:03:34,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-25 08:03:35,136 [I] DB_PATCHES have already been applied
I20251025 08:03:35.142620 28176 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-admin:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskXgZzmG/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251025 08:03:35.142735 28176 mini_ranger.cc:193] Using host: 127.27.132.20
I20251025 08:03:35.146624 28176 mini_ranger.cc:227] Ranger admin URL: http://127.27.132.20:53089
Oct 25, 2025 8:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 25, 2025 8:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 25, 2025 8:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-admin/ews/webapp
Oct 25, 2025 8:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 25, 2025 8:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 25, 2025 8:03:35 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 25, 2025 8:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-admin/ews/webapp].
Oct 25, 2025 8:03:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 25, 2025 8:03:36 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-53089"]
Oct 25, 2025 8:03:36 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 25, 2025 8:03:36 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 25, 2025 8:03:36 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 25, 2025 8:03:36 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 25, 2025 8:03:36 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 25, 2025 8:03:37 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Oct 25, 2025 8:03:37 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 25, 2025 8:03:37 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Oct 25, 2025 8:03:37 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20251025 08:03:37.372695 28176 mini_ranger.cc:160] Time spent starting Ranger: real 9.901s user 0.000s sys 0.010s
Oct 25, 2025 8:03:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 25, 2025 8:03:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Oct 25, 2025 8:03:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 25, 2025 8:03:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 25, 2025 8:03:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Oct 25, 2025 8:03:57 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-53089"]
I20251025 08:03:57.890749 28176 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.132.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.132.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.132.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/rangerkms_127.27.132.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.132.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/rangerkms_127.27.132.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.132.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.132.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/HTTP_127.27.132.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.132.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/HTTP_127.27.132.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Oct 25 08:03:57 dist-test-slave-lq8l krb5kdc[7166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379437, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251025 08:03:57.975977 28176 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251025 08:03:57.987196 28176 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251025 08:03:58.095677 28176 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251025 08:03:58.095784 28176 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-kms
I20251025 08:03:58.095829 28176 mini_ranger_kms.cc:227] Using postgres at 127.27.132.20:41335
2025-10-25 08:03:58,268 [I] DB FLAVOR :POSTGRES
2025-10-25 08:03:58,268 [I] --------- Verifying Ranger DB connection ---------
2025-10-25 08:03:58,268 [I] Checking connection
2025-10-25 08:03:58,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-10-25 08:03:58,594 [I] connection success
2025-10-25 08:03:58,595 [I] --------- Verifying Ranger DB tables ---------
2025-10-25 08:03:58,595 [I] Verifying table ranger_masterkey in database rangerkms
2025-10-25 08:03:58,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-10-25 08:03:58,897 [I] Table ranger_masterkey does not exist in database rangerkms
2025-10-25 08:03:58,897 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-25 08:03:58,897 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-10-25 08:03:58,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.132.20:41335/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-10-25 08:03:59,231 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251025 08:03:59.703258 28176 mini_ranger_kms.cc:310] Created kms service
I20251025 08:03:59.921650 28176 mini_ranger_kms.cc:326] Created kudu user
I20251025 08:03:59.963754 28176 mini_ranger_kms.cc:343] Created rangerkms user
Oct 25 08:03:59 dist-test-slave-lq8l krb5kdc[7166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379437, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.132.20@KRBTEST.COM
I20251025 08:04:00.336849 28176 mini_ranger_kms.cc:384] Added ranger policy
I20251025 08:04:00.337074 28176 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-kms:/tmp/dist-test-taskXgZzmG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskXgZzmG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251025 08:04:00.337100 28176 mini_ranger_kms.cc:240] Using host: 127.27.132.20
I20251025 08:04:00.340341 28176 mini_ranger_kms.cc:276] Ranger KMS PID: 7584
I20251025 08:04:00.340451 28176 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.132.20:59213
Oct 25, 2025 8:04:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 25, 2025 8:04:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 25, 2025 8:04:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Oct 25, 2025 8:04:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-kms/ews/webapp] .....
Oct 25, 2025 8:04:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-kms/ews/webapp].
Oct 25, 2025 8:04:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Oct 25, 2025 8:04:01 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-59213"]
Oct 25, 2025 8:04:01 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 25, 2025 8:04:01 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 25, 2025 8:04:01 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 25, 2025 8:04:02 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 25 08:04:02 dist-test-slave-lq8l krb5kdc[7166](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761379442, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.132.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:04:02.558845 28176 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.583s user 0.021s sys 0.134s
I20251025 08:04:02.558979 28176 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251025 08:04:02.559072 28176 mini_ranger_kms.cc:401] 127.27.132.20:59213/kms/v1/keys
Oct 25 08:04:02 dist-test-slave-lq8l krb5kdc[7166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379442, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.132.20@KRBTEST.COM for HTTP/127.27.132.20@KRBTEST.COM
Oct 25 08:04:03 dist-test-slave-lq8l krb5kdc[7166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379442, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.132.20@KRBTEST.COM for HTTP/127.27.132.20@KRBTEST.COM
Oct 25 08:04:05 dist-test-slave-lq8l krb5kdc[7166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:04:05.666702 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:42133
--webserver_interface=127.27.132.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.132.62:42133
--ranger_config_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.132.20:59213/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:04:05.774590 7637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:04:05.774871 7637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:04:05.774924 7637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:04:05.778231 7637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 08:04:05.778301 7637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:04:05.778340 7637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:04:05.778360 7637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 08:04:05.778378 7637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 08:04:05.782913 7637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.132.20:59213/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.132.62:42133
--ranger_config_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.132.62:42133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.132.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:04:05.784065 7637 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:04:05.784945 7637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:04:05.791210 7643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:04:05.791136 7645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:04:05.791373 7637 server_base.cc:1047] running on GCE node
W20251025 08:04:05.791158 7642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:04:05.792038 7637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:04:05.793105 7637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:04:05.794255 7637 hybrid_clock.cc:648] HybridClock initialized: now 1761379445794239 us; error 46 us; skew 500 ppm
Oct 25 08:04:05 dist-test-slave-lq8l krb5kdc[7166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379445, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:04:05.797145 7637 init.cc:377] Logged in from keytab as kudu/127.27.132.62@KRBTEST.COM (short username kudu)
I20251025 08:04:05.798252 7637 webserver.cc:492] Webserver started at http://127.27.132.62:46183/ using document root <none> and password file <none>
I20251025 08:04:05.798797 7637 fs_manager.cc:362] Metadata directory not provided
I20251025 08:04:05.798851 7637 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:04:05.799057 7637 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 25 08:04:05 dist-test-slave-lq8l krb5kdc[7166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379445, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.62@KRBTEST.COM for HTTP/127.27.132.20@KRBTEST.COM
I20251025 08:04:05.846138 7637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/data/instance:
uuid: "4e27ae2d34b947fe8fc151272eefa381"
format_stamp: "Formatted at 2025-10-25 08:04:05 on dist-test-slave-lq8l"
server_key: "cdfde5cd8c0e12a1946655dc05045e0f"
server_key_iv: "dc0591a99bb7bb6024878cc3a95129be"
server_key_version: "kuduclusterkey@0"
I20251025 08:04:05.846793 7637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/wal/instance:
uuid: "4e27ae2d34b947fe8fc151272eefa381"
format_stamp: "Formatted at 2025-10-25 08:04:05 on dist-test-slave-lq8l"
server_key: "cdfde5cd8c0e12a1946655dc05045e0f"
server_key_iv: "dc0591a99bb7bb6024878cc3a95129be"
server_key_version: "kuduclusterkey@0"
I20251025 08:04:05.851543 7637 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20251025 08:04:05.868233 7652 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:04:05.869412 7637 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251025 08:04:05.869539 7637 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/wal
uuid: "4e27ae2d34b947fe8fc151272eefa381"
format_stamp: "Formatted at 2025-10-25 08:04:05 on dist-test-slave-lq8l"
server_key: "cdfde5cd8c0e12a1946655dc05045e0f"
server_key_iv: "dc0591a99bb7bb6024878cc3a95129be"
server_key_version: "kuduclusterkey@0"
I20251025 08:04:05.869657 7637 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:04:05.897269 7637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:04:05.900461 7637 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:04:05.900653 7637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:04:05.908401 7704 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.62:42133 every 8 connection(s)
I20251025 08:04:05.908402 7637 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.62:42133
I20251025 08:04:05.909531 7637 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/data/info.pb
I20251025 08:04:05.912442 7705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 08:04:05.913915 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 7637
I20251025 08:04:05.914068 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/wal/instance
I20251025 08:04:05.914315 28176 external_mini_cluster.cc:1468] Setting key e7d7cfe7a624388bbe4c7ff62f2e7425
I20251025 08:04:05.918293 7705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381: Bootstrap starting.
I20251025 08:04:05.920818 7705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381: Neither blocks nor log segments found. Creating new log.
I20251025 08:04:05.921784 7705 log.cc:826] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381: Log is configured to *not* fsync() on all Append() calls
Oct 25 08:04:05 dist-test-slave-lq8l krb5kdc[7166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:04:05.923923 7705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381: No bootstrap required, opened a new log
I20251025 08:04:05.927510 7705 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e27ae2d34b947fe8fc151272eefa381" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42133 } }
I20251025 08:04:05.927793 7705 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 08:04:05.927852 7705 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e27ae2d34b947fe8fc151272eefa381, State: Initialized, Role: FOLLOWER
I20251025 08:04:05.928400 7705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e27ae2d34b947fe8fc151272eefa381" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42133 } }
I20251025 08:04:05.928548 7705 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251025 08:04:05.928612 7705 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251025 08:04:05.928701 7705 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [term 0 FOLLOWER]: Advancing to term 1
I20251025 08:04:05.928859 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:05.915588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.27.132.62:42133)
1025 08:04:05.915991 (+ 403us) server_negotiation.cc:207] Beginning negotiation
1025 08:04:05.916000 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1025 08:04:05.916035 (+ 35us) server_negotiation.cc:408] Connection header received
1025 08:04:05.916556 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:04:05.916573 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:04:05.916868 (+ 295us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:04:05.917148 (+ 280us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:04:05.918265 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:04:05.919080 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:04:05.919977 (+ 897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:04:05.920180 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:04:05.922977 (+ 2797us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:04:05.923013 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:04:05.923026 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:04:05.923061 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:04:05.925825 (+ 2764us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:04:05.926351 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:04:05.926357 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:04:05.926363 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:04:05.926457 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:04:05.926717 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:04:05.926721 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:04:05.926723 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:04:05.927214 (+ 491us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:04:05.927491 (+ 277us) server_negotiation.cc:1036] Waiting for connection context
1025 08:04:05.927784 (+ 293us) server_negotiation.cc:300] Negotiation successful
1025 08:04:05.928056 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20251025 08:04:05.929638 7705 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e27ae2d34b947fe8fc151272eefa381" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42133 } }
I20251025 08:04:05.929998 7705 leader_election.cc:304] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e27ae2d34b947fe8fc151272eefa381; no voters:
I20251025 08:04:05.930295 7705 leader_election.cc:290] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251025 08:04:05.930388 7710 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 08:04:05.930601 7710 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [term 1 LEADER]: Becoming Leader. State: Replica: 4e27ae2d34b947fe8fc151272eefa381, State: Running, Role: LEADER
I20251025 08:04:05.930989 7710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e27ae2d34b947fe8fc151272eefa381" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42133 } }
I20251025 08:04:05.931600 7705 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 08:04:05.932372 7712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e27ae2d34b947fe8fc151272eefa381. Latest consensus state: current_term: 1 leader_uuid: "4e27ae2d34b947fe8fc151272eefa381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e27ae2d34b947fe8fc151272eefa381" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42133 } } }
I20251025 08:04:05.932600 7712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [sys.catalog]: This master's current role is: LEADER
I20251025 08:04:05.932789 7711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e27ae2d34b947fe8fc151272eefa381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e27ae2d34b947fe8fc151272eefa381" member_type: VOTER last_known_addr { host: "127.27.132.62" port: 42133 } } }
I20251025 08:04:05.932896 7711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381 [sys.catalog]: This master's current role is: LEADER
I20251025 08:04:05.933084 7719 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 08:04:05.936372 7719 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 08:04:05.936909 7705 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251025 08:04:05.946906 7719 catalog_manager.cc:1357] Generated new cluster ID: 81284d21bd4a4bd69ebb2098d0054ba0
I20251025 08:04:05.947145 7719 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 08:04:05.959425 7719 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 08:04:05.960834 7719 catalog_manager.cc:1514] Loading token signing keys...
I20251025 08:04:05.974363 7719 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4e27ae2d34b947fe8fc151272eefa381: Generated new TSK 0
I20251025 08:04:05.975453 7719 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:04:06.726332 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.132.20:59213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.1:0
--local_ip_for_outbound_sockets=127.27.132.1
--webserver_interface=127.27.132.1
--webserver_port=0
--tserver_master_addrs=127.27.132.62:42133
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251025 08:04:06.887897 7751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:04:06.888262 7751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:04:06.888365 7751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:04:06.893245 7751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:04:06.893350 7751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:04:06.893489 7751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.1
I20251025 08:04:06.900045 7751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.132.20:59213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.132.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:42133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.1
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:04:06.901556 7751 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:04:06.902904 7751 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:04:06.911195 7759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:04:06.911526 7761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:04:06.911671 7758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:04:06.912307 7751 server_base.cc:1047] running on GCE node
I20251025 08:04:06.912717 7751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:04:06.913411 7751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:04:06.914620 7751 hybrid_clock.cc:648] HybridClock initialized: now 1761379446914597 us; error 43 us; skew 500 ppm
Oct 25 08:04:06 dist-test-slave-lq8l krb5kdc[7166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379446, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:04:06.918147 7751 init.cc:377] Logged in from keytab as kudu/127.27.132.1@KRBTEST.COM (short username kudu)
I20251025 08:04:06.919565 7751 webserver.cc:492] Webserver started at http://127.27.132.1:42231/ using document root <none> and password file <none>
I20251025 08:04:06.920341 7751 fs_manager.cc:362] Metadata directory not provided
I20251025 08:04:06.920421 7751 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:04:06.920730 7751 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 25 08:04:06 dist-test-slave-lq8l krb5kdc[7166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379446, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for HTTP/127.27.132.20@KRBTEST.COM
I20251025 08:04:06.935772 7751 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/data/instance:
uuid: "99da008efb3a44b6a97eab8e220c5345"
format_stamp: "Formatted at 2025-10-25 08:04:06 on dist-test-slave-lq8l"
server_key: "9456034c824a5278bc6468b225f2bdbb"
server_key_iv: "4cb3347da24c131d67beea7829081f90"
server_key_version: "kuduclusterkey@0"
I20251025 08:04:06.936544 7751 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance:
uuid: "99da008efb3a44b6a97eab8e220c5345"
format_stamp: "Formatted at 2025-10-25 08:04:06 on dist-test-slave-lq8l"
server_key: "9456034c824a5278bc6468b225f2bdbb"
server_key_iv: "4cb3347da24c131d67beea7829081f90"
server_key_version: "kuduclusterkey@0"
I20251025 08:04:06.944965 7751 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.000s
I20251025 08:04:06.984634 7768 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:04:06.986192 7751 fs_manager.cc:730] Time spent opening block manager: real 0.023s user 0.001s sys 0.000s
I20251025 08:04:06.986344 7751 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/wal
uuid: "99da008efb3a44b6a97eab8e220c5345"
format_stamp: "Formatted at 2025-10-25 08:04:06 on dist-test-slave-lq8l"
server_key: "9456034c824a5278bc6468b225f2bdbb"
server_key_iv: "4cb3347da24c131d67beea7829081f90"
server_key_version: "kuduclusterkey@0"
I20251025 08:04:06.986469 7751 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:04:07.009124 7751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:04:07.013093 7751 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:04:07.013371 7751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:04:07.014098 7751 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:04:07.015267 7751 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:04:07.015398 7751 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:04:07.015524 7751 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:04:07.015606 7751 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:04:07.026551 7751 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.1:45377
I20251025 08:04:07.027922 7751 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/data/info.pb
I20251025 08:04:07.030880 7883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.1:45377 every 8 connection(s)
I20251025 08:04:07.037250 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 7751
I20251025 08:04:07.037422 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-0/wal/instance
I20251025 08:04:07.037703 28176 external_mini_cluster.cc:1468] Setting key be7c2966a8607852964e42980fd89791
Oct 25 08:04:07 dist-test-slave-lq8l krb5kdc[7166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379446, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.1@KRBTEST.COM for kudu/127.27.132.62@KRBTEST.COM
I20251025 08:04:07.049842 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:07.033052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.1:48443)
1025 08:04:07.034893 (+ 1841us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:07.034940 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1025 08:04:07.035948 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 08:04:07.036949 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1025 08:04:07.036961 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1025 08:04:07.037424 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
1025 08:04:07.039357 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:04:07.039376 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:04:07.040503 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1025 08:04:07.040507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1025 08:04:07.041106 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1025 08:04:07.041116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1025 08:04:07.041349 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:04:07.042479 (+ 1130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1025 08:04:07.042520 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1025 08:04:07.044230 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1025 08:04:07.046623 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:04:07.046629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:04:07.046643 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:04:07.046996 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:04:07.047306 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1025 08:04:07.047309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1025 08:04:07.047312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1025 08:04:07.047475 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1025 08:04:07.047973 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1025 08:04:07.047980 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1025 08:04:07.048335 (+ 355us) client_negotiation.cc:769] Sending connection context
1025 08:04:07.048583 (+ 248us) client_negotiation.cc:240] Negotiation successful
1025 08:04:07.048891 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1620,"thread_start_us":124,"threads_started":1}
I20251025 08:04:07.050674 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:07.034206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.132.1:48443 (local address 127.27.132.62:42133)
1025 08:04:07.036424 (+ 2218us) server_negotiation.cc:207] Beginning negotiation
1025 08:04:07.036429 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1025 08:04:07.036445 (+ 16us) server_negotiation.cc:408] Connection header received
1025 08:04:07.036491 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1025 08:04:07.036494 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1025 08:04:07.036544 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1025 08:04:07.036630 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1025 08:04:07.039533 (+ 2903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:04:07.040355 (+ 822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1025 08:04:07.041879 (+ 1524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1025 08:04:07.042100 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1025 08:04:07.044417 (+ 2317us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1025 08:04:07.044438 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1025 08:04:07.044442 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1025 08:04:07.044473 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1025 08:04:07.046461 (+ 1988us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:04:07.047145 (+ 684us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:04:07.047149 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:04:07.047150 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:04:07.047203 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1025 08:04:07.047601 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1025 08:04:07.047604 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1025 08:04:07.047606 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1025 08:04:07.047807 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1025 08:04:07.047909 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1025 08:04:07.050351 (+ 2442us) server_negotiation.cc:300] Negotiation successful
1025 08:04:07.050495 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2122,"thread_start_us":71,"threads_started":1}
I20251025 08:04:07.063784 7884 heartbeater.cc:344] Connected to a master server at 127.27.132.62:42133
I20251025 08:04:07.064113 7884 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.27.132.2@KRBTEST.COM; defaulting to no policy
I20251025 08:04:07.064852 7884 heartbeater.cc:507] Master 127.27.132.62:42133 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.132.2@KRBTEST.COM" created.
I20251025 08:04:07.066817 7669 ts_manager.cc:194] Registered new tserver with Master: 99da008efb3a44b6a97eab8e220c5345 (127.27.132.1:45377)
I20251025 08:04:07.068513 7669 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.132.1@KRBTEST.COM'} at 127.27.132.1:48443
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.27.132.62 from keytab /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.132.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.132.2@KRBTEST.COM" created.
I20251025 08:04:07.111423 7743 server.cc:273] Received an EOF from the subprocess
F20251025 08:04:07.112490 7730 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7fb51b21fdcd google::LogMessage::Fail() at ??:0
@ 0x7fb51b223b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb51b21f7cc google::LogMessage::Flush() at ??:0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.132.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab.
I20251025 08:04:07.120878 28176 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
/tmp/dist-test-taskXgZzmG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.132.20:59213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.132.2:0
--local_ip_for_outbound_sockets=127.27.132.2
--webserver_interface=127.27.132.2
--webserver_port=0
--tserver_master_addrs=127.27.132.62:42133
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.132.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
@ 0x7fb51b220f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb51ced52d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7fb51ced20d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7fb51ced6005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x558aac153bea std::function<>::operator()() at ??:0
@ 0x7fb51c679598 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb51cf116db start_thread at ??:0
@ 0x7fb51a3d671f clone at ??:0
W20251025 08:04:07.293177 7895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 08:04:07.293525 7895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 08:04:07.293607 7895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 08:04:07.297575 7895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251025 08:04:07.297662 7895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 08:04:07.297755 7895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.132.2
I20251025 08:04:07.302345 7895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.132.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.132.20:59213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.132.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.132.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.132.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.132.62:42133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.132.2
--log_dir=/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type DEBUG
built by None at 25 Oct 2025 07:43:14 UTC on 24a791456cd2
build id 8706
I20251025 08:04:07.303587 7895 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 08:04:07.304497 7895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 08:04:07.311468 7909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:04:07.311468 7912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 08:04:07.312083 7910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 08:04:07.312155 7895 server_base.cc:1047] running on GCE node
I20251025 08:04:07.312638 7895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 08:04:07.313374 7895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 08:04:07.314559 7895 hybrid_clock.cc:648] HybridClock initialized: now 1761379447314534 us; error 54 us; skew 500 ppm
Oct 25 08:04:07 dist-test-slave-lq8l krb5kdc[7166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379447, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251025 08:04:07.318331 7895 init.cc:377] Logged in from keytab as kudu/127.27.132.2@KRBTEST.COM (short username kudu)
I20251025 08:04:07.319860 7895 webserver.cc:492] Webserver started at http://127.27.132.2:43671/ using document root <none> and password file <none>
I20251025 08:04:07.320619 7895 fs_manager.cc:362] Metadata directory not provided
I20251025 08:04:07.320699 7895 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 08:04:07.320945 7895 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 25 08:04:07 dist-test-slave-lq8l krb5kdc[7166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761379447, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.132.2@KRBTEST.COM for HTTP/127.27.132.20@KRBTEST.COM
I20251025 08:04:07.338755 7895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/data/instance:
uuid: "f08c744704d04ea0b47a7e417f5ea73d"
format_stamp: "Formatted at 2025-10-25 08:04:07 on dist-test-slave-lq8l"
server_key: "7a32bc0a89f644f13e7c7ea70f1fb5a6"
server_key_iv: "448c0a560ab747ef3c461b71b465fda3"
server_key_version: "kuduclusterkey@0"
I20251025 08:04:07.339514 7895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance:
uuid: "f08c744704d04ea0b47a7e417f5ea73d"
format_stamp: "Formatted at 2025-10-25 08:04:07 on dist-test-slave-lq8l"
server_key: "7a32bc0a89f644f13e7c7ea70f1fb5a6"
server_key_iv: "448c0a560ab747ef3c461b71b465fda3"
server_key_version: "kuduclusterkey@0"
I20251025 08:04:07.344484 7895 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251025 08:04:07.361886 7919 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 08:04:07.363907 7895 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20251025 08:04:07.364058 7895 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/wal
uuid: "f08c744704d04ea0b47a7e417f5ea73d"
format_stamp: "Formatted at 2025-10-25 08:04:07 on dist-test-slave-lq8l"
server_key: "7a32bc0a89f644f13e7c7ea70f1fb5a6"
server_key_iv: "448c0a560ab747ef3c461b71b465fda3"
server_key_version: "kuduclusterkey@0"
I20251025 08:04:07.364192 7895 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 08:04:07.381696 7895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 08:04:07.384886 7895 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 08:04:07.385083 7895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 08:04:07.385699 7895 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 08:04:07.386605 7895 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 08:04:07.386653 7895 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 08:04:07.386698 7895 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 08:04:07.386729 7895 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 08:04:07.396374 7895 rpc_server.cc:307] RPC server started. Bound to: 127.27.132.2:41351
I20251025 08:04:07.396682 8032 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.132.2:41351 every 8 connection(s)
I20251025 08:04:07.397418 7895 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/data/info.pb
I20251025 08:04:07.401005 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:07.399631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.2:44093)
1025 08:04:07.400132 (+ 501us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:07.400254 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":100,"threads_started":1}
W20251025 08:04:07.401652 8033 heartbeater.cc:646] Failed to heartbeat to 127.27.132.62:42133 (0 consecutive failures): Network error: Failed to ping master at 127.27.132.62:42133: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
I20251025 08:04:07.402290 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:07.402042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.2:54723)
1025 08:04:07.402169 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:07.402223 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20251025 08:04:07.402900 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:07.402689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.2:56179)
1025 08:04:07.402797 (+ 108us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:07.402842 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20251025 08:04:07.403050 8033 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251025 08:04:07.404430 28176 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskXgZzmG/build/debug/bin/kudu as pid 7895
I20251025 08:04:07.404551 28176 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskXgZzmG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761379300881738-28176-0/minicluster-data/ts-1/wal/instance
I20251025 08:04:07.404824 28176 external_mini_cluster.cc:1468] Setting key 50189620a3dc6edb1456548d25359f8c
I20251025 08:04:08.073343 7884 heartbeater.cc:499] Master 127.27.132.62:42133 was elected leader, sending a full tablet report...
I20251025 08:04:08.074352 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:08.073899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.1:53799)
1025 08:04:08.074168 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:08.074242 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
W20251025 08:04:08.074611 7884 heartbeater.cc:646] Failed to heartbeat to 127.27.132.62:42133 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
I20251025 08:04:08.404192 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:08.403737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.2:60309)
1025 08:04:08.404025 (+ 288us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:08.404082 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
I20251025 08:04:09.075915 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:09.075405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.1:48017)
1025 08:04:09.075721 (+ 316us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:09.075789 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20251025 08:04:09.405604 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:09.405113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.2:36633)
1025 08:04:09.405411 (+ 298us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:09.405480 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20251025 08:04:10.077239 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:10.076759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.1:39335)
1025 08:04:10.077074 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:10.077139 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":87,"threads_started":1}
I20251025 08:04:10.407027 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:10.406606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.2:55647)
1025 08:04:10.406871 (+ 265us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:10.406929 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20251025 08:04:11.078651 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:11.078165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.1:37201)
1025 08:04:11.078470 (+ 305us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:11.078545 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":103,"threads_started":1}
I20251025 08:04:11.408252 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:11.407828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.2:35295)
1025 08:04:11.408098 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:11.408154 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
I20251025 08:04:12.080175 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:12.079689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.1:53047)
1025 08:04:12.080012 (+ 323us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:12.080076 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":88,"threads_started":1}
I20251025 08:04:12.409627 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1025 08:04:12.409151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.132.62:42133 (local address 127.27.132.2:40669)
1025 08:04:12.409454 (+ 303us) negotiation.cc:107] Waiting for socket to connect
1025 08:04:12.409520 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.132.62:42133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
F20251025 08:04:12.448457 28176 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1761379452 (unix time) try "date -d @1761379452" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800006e10) received by PID 28176 (TID 0x7fee4b183d80) from PID 28176; stack trace: ***
@ 0x7fee56e1d980 (unknown) at ??:0
@ 0x7fee54455fb7 gsignal at ??:0
@ 0x7fee54457921 abort at ??:0
@ 0x7fee556c3dcd google::LogMessage::Fail() at ??:0
@ 0x7fee556c7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fee556c37cc google::LogMessage::Flush() at ??:0
@ 0x7fee556c4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fee5844bcab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7fee584489c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x5636be77292f kudu::SecurityITest::StartCluster() at ??:0
@ 0x5636be75a792 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7fee566d30ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fee566c7bdb testing::Test::Run() at ??:0
@ 0x7fee566c7d9d testing::TestInfo::Run() at ??:0
@ 0x7fee566c8377 testing::TestSuite::Run() at ??:0
@ 0x7fee566c877c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7fee566d360d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fee566c7e63 testing::UnitTest::Run() at ??:0
@ 0x5636be7aac7e RUN_ALL_TESTS() at ??:0
@ 0x5636be7a9e6b main at ??:0
@ 0x7fee54438bf7 __libc_start_main at ??:0
@ 0x5636be74206a _start at ??:0
2025-10-25 08:04:12.639 UTC [7512] FATAL: terminating connection due to unexpected postmaster exit
2025-10-25 08:04:12.639 UTC [7615] FATAL: terminating connection due to unexpected postmaster exit
2025-10-25 08:04:12.641 UTC [7511] FATAL: terminating connection due to unexpected postmaster exit
2025-10-25 08:04:12.642 UTC [7508] FATAL: terminating connection due to unexpected postmaster exit
2025-10-25 08:04:12.643 UTC [7507] FATAL: terminating connection due to unexpected postmaster exit
2025-10-25 08:04:12.647 UTC [7509] FATAL: terminating connection due to unexpected postmaster exit